Long context: what a million tokens can and can't do for you

Context windows keep growing. Two million tokens is now a headline spec, and every frontier model happily accepts entire codebases, book series, or document archives in a single prompt. The demos look like magic: a model pulls a specific detail out of a million-token haystack without breaking a sweat.

But that demo measures one thing — finding a fact. The research on long-context models draws a sharp line between that party trick and something far more valuable: reasoning over everything in the window. And on that second skill, the million-token models are much less impressive than their marketing suggests.

What "needle in a haystack" actually measures#

The classic needle-in-a-haystack (NIAH) test, popularized by Greg Kamradt in 2023, buries one sentence in a huge pile of irrelevant text and asks the model to retrieve it. On this test, frontier models are genuinely excellent. Google DeepMind reported Gemini 1.5 Pro hitting over 99.7% recall on a single buried fact at 1 million tokens — effectively perfect.

Three caveats, each important:

1. One needle is the easy case. The same Google report tested a variant with 100 facts hidden instead of one. Gemini 1.5 Pro's recall fell to roughly 60% at 1M tokens; GPT-4 Turbo dropped to about 50% at its 128K limit. Real work is rarely single-needle retrieval — it usually involves gathering, comparing, or counting multiple scattered facts.

1. NIAH is a retrieval test, not an understanding test. Finding the one sentence that literally contains the answer requires matching keywords, not comprehending the documents. The HELMET evaluation work showed explicitly that NIAH performance does not predict how well a model handles downstream long-context tasks.

1. The score is fragile to the prompt itself. When Anthropic tested Claude 2.1 on a 200K-token haystack, adding a single sentence to the prompt lifted its needle score from 27% to 98%. That's not a model getting smarter mid-benchmark — it's evidence the metric is measuring something brittle, and easily gamed by prompt engineering.

None of this means NIAH is worthless. It's a useful floor: if a model can't do it, don't trust it with long inputs at all. But a perfect NIAH score is not proof the model can reason across a million tokens. It proves it can find a distinctive sentence.

Reasoning over the haystack is a different sport#

The moment you ask a model to synthesize, compare, aggregate, or chain facts across a long context, performance collapses much earlier than the headline window size suggests. This is the core finding across several independent benchmarks:

• RULER (NVIDIA, COLM 2024). The RULER authors extended NIAH into four task families — retrieval with multiple needles, multi-hop tracing (following chains of variable assignments), aggregation (extracting common words across the context), and QA with distractors. They evaluated 17 long-context models claiming windows of 32K or more. Despite nearly perfect vanilla NIAH scores, almost all models showed large degradation as length increased, and only about half could handle just 32K tokens to a satisfactory threshold. Nearly every model fell below the bar well before its claimed context length.

• NoLiMa. Where RULER makes retrieval harder by multiplying needles, NoLiMa (2025) makes it harder by removing the lexical crutch: the question and the relevant passage share minimal word overlap, forcing the model to infer associations rather than match strings. Around 10 of the 12 tested models dropped below 50% of their short-context baseline at 32K tokens in that setting; GPT-4o fell from 99.3% to 69.7%. In practice, real queries usually need semantic matching, not literal matching — which is exactly the mode that degrades.

• BABILong. Embedding bAbI reasoning tasks inside haystacks up to a million tokens produced the bleakest headline: once reasoning is required, models effectively use only about 10–20% of their context, with some degrading past just 13K tokens.

• LongBench v2. A harder, more realistic multitask benchmark (503 multiple-choice questions over contexts from 8K to 2M tokens) showed models winning only narrowly: GPT-4o reached 50.1% where human experts under time pressure scored 53.7%. And revealingly, some models performed better with a 32K retrieval slice than with their full 128K window — more context can actively hurt.

• LongBench Pro. Evaluating 46 models, the authors found effective context length is typically shorter than claimed length, with pronounced cross-lingual misalignment — performance in one language doesn't transfer cleanly to another at long contexts. Long-context optimization contributed more than raw parameter scaling.

The pattern is consistent: effective context length is a fraction of advertised length. One independent summary put the rule of thumb at roughly 25–50% of the claimed window — for example, a 128K-claim model being reliably usable to about 32–64K. Treat the headline number as a ceiling for "can accept," not a promise of "can use."

Why the gap exists#

This isn't one bug; it's several compounding ones:

• Lost in the middle. Liu et al.'s 2023 "Lost in the Middle" study found a U-shaped accuracy curve: models answer best when the relevant information sits at the start or end of the context, and substantially worse when it's buried in the middle — accuracy dropping 20+ percentage points. Information at the edges is privileged.
• Attention economics. Softmax attention is a zero-sum game: weights sum to 1, so every extra token dilutes the focus available for relevant ones. Studies of attention sinks found the first few tokens absorbing 15–40% of attention mass regardless of content — pure overhead that never goes away as context grows.
• Length itself is a tax. One 2025 analysis found performance degrading 13.9–85% with longer inputs even when all irrelevant tokens were masked and retrieval was perfect. Longer isn't just noisier; it appears to impose cognitive load independent of content quality.
• Hallucination pressure. The RULER authors observed that as contexts grow, models increasingly fall back on parametric knowledge — answering from training memory instead of the provided text. The context becomes decoration.

What this means in practice#

The research doesn't say long context is useless. It says the skill is narrower than the spec sheet implies. A few rules that follow from the evidence:

• Prefer retrieval over dumping. When the corpus exceeds what you'll actually use, retrieval (RAG) still wins: LongBench v2 showed a 32K retrieval slice beating a 128K full-window pass for some models. RAG plus a moderately sized window beats "just paste everything" on both accuracy and cost.
• Put key instructions and facts at the edges. The lost-in-the-middle effect is reproducible. If you control the prompt, place the question and critical facts at the beginning or end — never trust the middle for the one thing that matters.
• Ask for synthesis, verify with retrieval. If you need the model to reason across long documents, check the answer: ask it to quote its sources. When the model can't point to where each claim came from, its synthesis over long context is exactly where the benchmarks say it's weakest.
• Many-shot in-context learning is the bright spot. One area where longer context genuinely helps is loading hundreds of examples into the prompt for in-context learning — that's retrieval-light, pattern-dense work, and it fits how the models actually use their windows.
• Assume the effective window, not the advertised one. Budget 25–50% of the claimed length for tasks that need reasoning, and test your specific task at your specific length rather than trusting the benchmark that came in the launch blog post.

The takeaway#

Needle-in-a-haystack is easy now — nearly every frontier model finds the needle. But the research consensus is that the haystack itself is still largely opaque to these models: they retrieve from it, they don't reliably reason over it. RULER, NoLiMa, BABILong, and LongBench all converge on the same point: the advertised context window measures capacity, and capacity is not capability.

That gap is narrowing — 2026 frontier models hold accuracy across their full windows far better than 2024's — but it hasn't closed. Until it does, the practical move is boring and effective: retrieve before you reason, structure your context deliberately, and verify answers against sources. A million tokens is a big desk. It doesn't make the model a fast reader.