Long-Context Evaluation: Chunking, Position Tests and Lost Details

Long inputs fail quietly. A model can read a 200-page report, answer fluently, and still skip the one clause that mattered. Audio and video add a second problem: the model may fill gaps with plausible guesses. This page gives you ways to test instead of hoping: how to word transcription and video tasks, how to place a known fact early, in the middle and late in a long input, how to detect details that a long output lost, and how to estimate size and cost before a big run.

The code uses the standard library and promptlab from Prompt Chaining: Stages, Fixed Schemas and the promptlab Helper. Every "model" in it is a toy provider: a deterministic stand-in so the test harness can be checked repeatably. Real models are less predictable, which is why you test them. Nothing printed by a toy says how a real model behaves. Chunking and hierarchical summaries were covered in Multimodal Prompt Design: Images, Scans, Tables and Long Documents; this page tests whether they work.

What you will learn

• How to write transcription, speaker-label and video-summary instructions that separate what the media shows from what you assume.
• How to build a needle-position test for early, middle and late facts, and how to read its result.
• How to detect lost details in long outputs with a must-keep list.
• How to treat context limits as per-model facts to look up, and how to estimate input size and cost roughly before a large run.
• How to evaluate a long-document question-answering prompt, including questions that have no answer.

Audio: transcription instructions

A transcript is evidence, so the instruction must say what a good transcript is. Decide before you ask: verbatim or cleaned up? Keep filler words? How to mark a word you cannot hear? What language and spelling? Without these choices the model picks for you, and different runs pick differently.

Weak prompt:

Transcribe this audio.

Better prompt:

Transcribe the attached audio in the language spoken. Keep the wording as spoken, but drop
filler sounds such as "um". Do not correct grammar. If a word is unclear, write [unclear]
and do not guess. Mark long silences and non-speech sounds as [silence] or [noise: type].
Put a timestamp [mm:ss] at the start of each paragraph.

What changed: the style is defined, a guess is replaced by a visible marker, and timestamps give you a way to check any line against the recording. After you get a transcript, listen to two or three passages and compare. Names, numbers and dates are where transcripts go wrong most often.

Speaker labels only when supported

Speaker diarisation means labelling who spoke each part. Not every tool or model does it, and a tool that does not support it may still produce labels that look real. So ask for labels only when you know your tool supports them, and treat them as guesses to check.

Provider note (as of 2026-10-12): Google's audio documentation lists speaker diarisation (example labels "Speaker 1", "Speaker 2"), timestamps, translation and detection of non-speech sounds as audio tasks, and says audio is downsampled and multi-channel audio is merged into one channel. Audio support in chat apps and with other providers varies; I did not confirm it for every provider. Check the current documentation for your tool.

Better prompt:

Label speakers as Speaker 1, Speaker 2 in order of first appearance. Do not use real names
unless a speaker says their own name in the audio; then note it once as "Speaker 2 (says: Amara)".
If you cannot tell two voices apart, write [speaker unclear] instead of choosing one.

Labels are not identities. A model cannot know that "Speaker 2" is the finance manager unless someone says so in the recording. Verify turn changes against the audio, and never send a label to a decision (who agreed to what) without a person listening.

Video: summaries with time references

A video summary without times cannot be checked. Ask for a short list of segments, each with a start time and one sentence of what happens, and keep what is seen or heard apart from what the model infers.

Better prompt:

Summarise the attached video as at most 6 segments. For each: [mm:ss-mm:ss], one sentence of
what is shown or said, and the source: "seen", "heard" or "both". Then list up to 3 questions the
video does not answer. If you cannot read on-screen text, say so instead of guessing it.

Open the video at three of the times and compare. Time references that drift or point to the wrong scene are a sign to shorten the clip or ask for fewer segments.

Provider note (as of 2026-10-12): Google's API documentation lists video as an input. I could not confirm video limits or support in chat apps and other providers. Check the current documentation for what your tool accepts.

Media evidence versus contextual assumptions

Media gives evidence; the surrounding text gives context. A caption such as "meeting in the Pune office" is context, not proof that the video shows Pune. Ask the model to label each claim, then check the labels.

Better prompt:

For each statement in your answer add one tag:
[MEDIA] directly visible or audible in the file (give the time or region),
[CONTEXT] taken from the text I provided,
[ASSUMPTION] your inference, not shown anywhere.
Do not present an [ASSUMPTION] as fact. Say "cannot be determined" where nothing supports an answer.

Example reply (illustrative; yours will differ):

[MEDIA] 00:42 A person points to a chart titled "Q3 returns".
[CONTEXT] The title says this is the Pune office review.
[ASSUMPTION] The speaker is the regional manager.
Cannot be determined: whether the chart values match the report; they are not readable.

What to verify: spot-check two [MEDIA] tags against the file (the model can mislabel), and move any [ASSUMPTION] you need into your own list of things to confirm.

Testing facts at early, middle and late positions

Providers agree on the placement advice: put the long material first and the question last, and label each document with a source. They do not promise that every fact in a long input is used equally. A research study of older models (Liu et al., 2023) found that accuracy was highest when the relevant fact was at the start or the end of the input and lowest in the middle. That is a direction tested on models of that time, not a law and not a number. Provider guidance also notes that finding one fact is easier than finding several. So for your model and your task: test.

The test is a needle test. Take a filler document, replace one record with a known fact at a chosen position, ask for it, and score the reply by exact match. Repeat with different fact values and positions. The toy reader below is a stand-in that is deliberately less reliable in the middle; it reproduces the lost-in-the-middle pattern so you can see what the harness shows, in direction only. A real model may do better, worse or differently, which is the reason to run the harness against it.

import random, zlib
from promptlab import Client, estimate_tokens, render

def make_doc(n=40):
    return [f"Record {i:02d}: branch {'ABCDEF'[i % 6]} handled {10 + (i * 7) % 23} requests; routine." for i in range(1, n + 1)]

def place(lines, frac, code):
    """Replace one record with the needle fact, at a relative position 0.0 (start) .. 1.0 (end)."""
    k = round(frac * (len(lines) - 1))
    out = lines[:]
    out[k] = f"Record {k + 1:02d}: the vault code is {code}."
    return out

QA = "Answer only from the document. If the answer is not there, say NOT FOUND.\n\n### DOCUMENT\n{doc}\n\n### QUESTION\n{q}"

def toy_reader(messages, **settings):
    """Stand-in: finds the needle less often in the middle (direction only, not a real model)."""
    text = messages[-1]["content"]
    lines = text.split("### DOCUMENT\n", 1)[1].split("\n\n### QUESTION", 1)[0].splitlines()
    hit = [i for i, line in enumerate(lines) if "vault code" in line]
    if not hit:
        return "NOT FOUND"
    pos = hit[0] / (len(lines) - 1)
    p_find = 0.95 - 0.45 * (1 - abs(2 * pos - 1))     # 0.95 at the ends, 0.50 in the middle
    if random.Random(zlib.crc32(text.encode())).random() < p_find:   # seeded from the prompt: repeatable
        return lines[hit[0]].split(" is ")[1].rstrip(".")
    return "NOT FOUND"

client = Client(toy_reader)
rng = random.Random(7)
base = make_doc()
print("document:", len(base), "records,", estimate_tokens("\n".join(base)), "tokens (rough)")
for name, frac in [("early", 0.1), ("middle", 0.5), ("late", 0.9)]:
    found = 0
    for _ in range(40):
        code = f"{rng.randrange(10000, 99999)}"
        doc = "\n".join(place(base, frac, code))
        reply = client.complete([{"role": "user", "content": render(QA, doc=doc, q="What is the vault code?")}]).text
        found += reply == code
    print(f"{name:6} (position {frac}): found {found}/40")

document: 40 records, 500 tokens (rough)
early  (position 0.1): found 35/40
middle (position 0.5): found 15/40
late   (position 0.9): found 38/40

Read it the way you would read a real run: 40 trials per position, the same question, only the position changes. The middle row is clearly lower; that is the planted behaviour of the toy. With a real model, expect noisy rows, so use enough trials and several fact values, and report the counts, not a single try. If the middle is weak for your model, shorten the input, move key material to the start or end, or retrieve the relevant parts first (a topic for the AI Engineer path).

When you test several facts at once, score each fact separately, because finding one needle does not show that the model found three.

Detecting lost details in long outputs

Summaries and long answers lose details without any error message. The fix is a must-keep list: facts, numbers and names you can write down from the source before you run the model, checked in code against the output. It cannot prove the summary is right, but it catches missing items cheaply.

REPORT = ["Summary of the quarter. Orders grew in most regions.",
          "Region North: order 48213 was refunded on 2026-03-14 after a damaged delivery.",
          "Region South: delivery times fell by two days.",
          "Region East: Lena Fischer joined as regional lead.",
          "Closing note. Next review is in six months."]
MUST = ["48213", "2026-03-14", "Lena Fischer"]

def toy_summary(paragraphs):
    """Stand-in summariser that keeps only the first and last paragraph, so middle details vanish."""
    return " ".join([paragraphs[0], paragraphs[-1]])

def lost_details(output, must=MUST):
    return [item for item in must if item.lower() not in output.lower()]

print("lost:", lost_details(toy_summary(REPORT)))
print("lost, full text:", lost_details(" ".join(REPORT)))

lost: ['48213', '2026-03-14', 'Lena Fischer']
lost, full text: []

The second line is a sanity check of the checker: the full text loses nothing. A match is only a string match, so a detail can be present and wrong; add a second check for values (the date is 2026-03-14, not another one). In the prompt, you can also ask for the detail to be kept: "Keep every order number, date and person's name from the source."

Context limits: do not assume one limit

A context window is the most a model can consider in one request. It covers your input, the output and any hidden reasoning. Different models have different windows, output caps and rules for images, audio and files, and these change when models are updated. So write no number into your design from memory.

Provider note (as of 2026-10-12): the providers' documentation says to look up the window and the maximum output on the page of the model you use (Google: check the Models page rather than relying on general figures). Reasoning tokens use up window space. Anthropic offers a free token-counting endpoint, whose result is an estimate, and says a newer tokenizer can produce noticeably more tokens for the same text than an older one, so count again for each model. Check the current documentation.

Comparing multimodal capabilities from provider documents

Before you choose a tool for audio, images, PDFs or video, fill a small table from each provider's current pages, and write the date you checked.

Estimating input size and cost before a large run

Run a small sample first, then multiply. estimate_tokens in promptlab is a rough estimator (one token per four characters). It is useful for comparing prompts and for catching a run that is ten times larger than you planned. It is not a tokenizer, and it is less accurate for non-English text. For a real budget use the provider's counter and the provider's usage data. The unit price below is an example price, not real.

PRICE_PER_MILLION = 0.50      # example price, not real
prompt = render(QA, doc="\n".join(place(make_doc(400), 0.5, "12345")), q="What is the vault code?")
per_call, calls = estimate_tokens(prompt), 3 * 40 * 5
total = per_call * calls
print(f"one call: {per_call} tokens (rough); {calls} calls: {total} tokens (rough)")
print(f"example cost: {total / 1_000_000 * PRICE_PER_MILLION:.2f} (example price, not real)")

one call: 5103 tokens (rough); 600 calls: 3061800 tokens (rough)
example cost: 1.53 (example price, not real)

The experiment above, with a document ten times longer and five times as many trials (200 per position), would send about three million (rough) input tokens. Seeing that number before you start is the point. Output tokens, hidden reasoning and any retry also count, and they are usually priced differently; check the provider's price sheet.

Where this shows up at work

Legal and compliance teams ask questions of long contracts. Support teams summarise call recordings. Researchers query long reports. In each case the same short routine applies: pick a few facts you can place and score, run them at several positions, keep a must-keep list for summaries, and estimate before you run the full batch.

Common mistakes

• Believing a large window means perfect recall. A fact can fit and still be missed. Test positions on your own model.
• Judging a position test from one try. Results are noisy; use many trials and different values.
• Trusting speaker labels and quoted times. Both are model output. Check them against the file.
• Reusing a token count across models. Tokenisers differ. Recount for each model.
• Scoring only answerable questions. Include questions with no answer, where the right reply is "not found".

Try it yourself

Run the page's blocks in order in one Python folder with promptlab.py beside them.

1. Easy. Write a transcription prompt for a two-person interview in Spanish that keeps the Spanish, marks unclear words and asks for no speaker names. Add a one-line English gloss of what it should return.
2. Medium. Rerun the needle test with a document of 80 records and positions 0.0, 0.25, 0.5, 0.75 and 1.0. Say which rows you would trust and why.
3. Hard. Evaluate a long-document question-answering prompt: three cases (fact early, fact in the middle, fact absent), a scorer that returns "correct", "abstained" or "wrong", and two readers: toy_reader and one that always invents an answer. Explain which reader you would accept.

Answers

Easy. A strong answer:

Transcribe the attached audio in Spanish, keeping the Spanish wording as spoken. Label the two
voices Interviewer and Guest; do not use real names. Write [unclear] for any word you cannot hear.
Add a timestamp [mm:ss] at each change of speaker.

Gloss: it returns a Spanish transcript with two role labels, no guessed names and timestamps. Pass checklist: language stated, unclear-word rule, no invented names, timestamps for checking.

Medium. Change make_doc(40) to make_doc(80), loop over the five positions and compare the counts. Pass checklist: same question and 40 or more trials per position; counts reported, not a single try; edge rows are trusted more than the middle only if the counts show it; you state that a toy reader's pattern is built in and a real model must be tested to know.

Hard.

def score(reply, expected):
    if reply == expected:
        return "correct"
    return "abstained" if reply == "NOT FOUND" else "wrong"

def toy_guesser(messages, **settings):
    """Stand-in that always invents an answer, even when the document has none."""
    return "99999"

cases = [("early", place(base, 0.1, "11111"), "11111"), ("middle", place(base, 0.5, "22222"), "22222"),
         ("absent", base, "NOT FOUND")]
for reader in (toy_reader, toy_guesser):
    row = [score(Client(reader).complete([{"role": "user", "content": render(QA, doc="\n".join(d), q="What is the vault code?")}]).text, want)
           for _, d, want in cases]
    print(reader.__name__, row)

toy_reader ['correct', 'correct', 'correct']
toy_guesser ['wrong', 'wrong', 'wrong']

Pass checklist: an absent-fact case exists; abstaining on it counts as correct, and an invented answer counts as wrong; the guesser is rejected even though it sometimes could match by chance; three cases are a demonstration, so a real evaluation needs many cases per position. The guesser is a stand-in, but the lesson is real: a prompt that never says "if it is not there, say so" invites made-up answers.

Summary

• Write transcription and video instructions that define style, mark unclear parts and require times you can check.
• Ask for speaker labels only where the tool supports them, and treat labels as guesses.
• Separate media evidence, supplied context and assumptions with tags, and spot-check the tags.
• Test early, middle and late placement on your own model, with many trials; use a must-keep list for lost details.
• Look up context limits and multimodal rules per model, and estimate size and cost roughly before large runs.

Next: Sampling and Reasoning Controls: Temperature, Top-p, Effort and Repeatability, where you learn which settings change variability and which do not change truth.