Every model launch arrives with the same ritual: a slide deck of benchmark scores, each one higher than the last, each one presented as proof of progress. MMLU this, HumanEval that. The numbers look rigorous. They aren't always.

Evaluating AI systems is itself a science — and right now, it's a science in crisis. Test questions leak into training data. Benchmarks that took years to build stop discriminating between models within months. And once a score becomes the thing labs optimize for, it quietly stops measuring what it was supposed to measure. This is the story of why evals break, and what the people who take measurement seriously do instead.

1. Contamination: the model has already seen the test#

Benchmark contamination is the simplest failure mode and the hardest to fully rule out. Modern models train on enormous scrapes of the internet, and benchmark questions live on the internet. When test items (or near-identical variants) end up in training data, a model can score well by memorization rather than capability. The score goes up; nothing was learned.

How big is the effect? In 2024, Microsoft Research released MMLU-CF, a contamination-free rebuild of the popular MMLU knowledge benchmark, using broader data sources, explicit decontamination rules, and — crucially — a closed-source test set to block malicious leakage. The results were sobering: GPT-4o, the strongest model tested, managed only 73.4% on the contamination-free test set in the 5-shot setting (71.9% zero-shot) — well below its headline MMLU score (arXiv:2412.15194).

A more recent study from Stanford and City University of Macau (July 2026) added an important nuance. Measuring contamination as a violation of "anchor-item invariance" — comparing performance on original items versus semantically equivalent paraphrases across 47 public models — the authors found that contamination is largely uniform: it inflates everyone's absolute scores but rarely reorders the leaderboard. The rank correlation between a standard leaderboard and a paraphrase-controlled one was 0.997, and only 3 of 188 model-by-benchmark cases showed differential contamination corroborated across references (arXiv:2609.02899).

That's a genuinely useful finding, but notice what it does not say. It says rankings are roughly trustworthy; it doesn't say the scores mean what the marketing claims they mean. A leaderboard can be ordered correctly and still be measuring memorization plus capability in an unknown ratio.

2. Saturation: benchmarks die within months#

Even a perfectly clean benchmark has a shelf life. The lifecycle is now predictable: a benchmark launches, models improve rapidly against it, scores bunch up near the ceiling, and the test stops telling us anything. MMLU went from 43.9% (GPT-3, 2020) to effectively saturated. GSM8K, once a serious math test, now sees near-perfect scores. HumanEval — writing short Python functions — has been largely abandoned for frontier comparisons in favor of harder successors like LiveCodeBench and SWE-bench. By early 2026, frontier models were clearing 90% on MMLU and pushing past human-expert performance on GPQA and AIME-style questions.

The treadmill keeps spinning. Each saturated benchmark spawns a harder one: MMLU → MMLU-Pro → GPQA → Humanity's Last Exam (a 2,500-question expert test published in Nature in 2026). And the new hard ones get conquered fast too. In October 2025, Epoch AI estimated that less than 70% of FrontierMath — a benchmark of original research-level math problems — was reliably within reach of any model run, despite headlines about 25–29% scores on individual runs. Then, by late summer 2026, Epoch's FrontierMath Tier 4 went from a 5% top score at launch to 98% and declared saturation in under 14 months.

Epoch's own answer to this churn is interesting: their game-puzzles benchmarks (chess puzzles, "mystery game" puzzles) and the Epoch Capabilities Index, which aggregates across math, coding, and gameplay rather than trusting any single test. As of 2026, frontier closed models sit at a 59% ceiling on the mystery game puzzles while open-weight models top out around 38% — scores with actual headroom, which is exactly the point.

3. Goodhart's law: when the measure becomes the target#

The economist Charles Goodhart observed in 1975 that "when a measure becomes a target, it ceases to be a good measure." AI benchmarking is arguably the largest-scale demonstration of this principle in history. Benchmark scores drive media coverage, which drives perception, which drives enterprise deals and investment. So labs optimize for benchmarks — sometimes at the direct expense of real capability.

The gaming happens at several levels:

  • Direct contamination — test items in training data, intentional or not.
  • Format overfitting — training on floods of MMLU-style multiple-choice questions or GSM8K-style word problems until the model masters the format rather than the underlying skill. Models learn to pattern-match answer choices; rephrase the question and scores can collapse.
  • Prompt and harness hacking — labs tune prompts per model to squeeze out points. Independent evaluator Artificial Analysis notes that Gemini 1.0 Ultra reportedly used custom 32-shot chain-of-thought prompting per MMLU topic to beat GPT-4 — effectively, when points are tight, you can "put the answer into the model." This is why AA runs every model through identical prompts itself and refuses to trust lab-reported numbers.
  • Hyperparameter selection — running many experiments and shipping the checkpoint that happened to score highest on the benchmarks.

The predictable endpoint of metric decay: a model can hit the 99th percentile on a coding benchmark and still struggle to write a simple original script. The correlation between the benchmark and real-world performance has decayed to zero or gone negative — the metric is now a hollow target.

4. Variance: the numbers are noisier than they look#

There's a subtler problem most score tables hide: randomness. A small multiple-choice eval run at a lab-recommended temperature can have "pretty enormous" variance on a reasoning model, in Artificial Analysis's words — a single run of a 4-question eval tells you almost nothing. AA reports running large numbers of repeats to dial in ±1 point at 95% confidence for its Intelligence Index, which multiplies evaluation cost enormously. (Its published "cost to run" figures assume one repeat; the real spend is far higher.)

Then there's the endpoint problem: when labs hand an evaluator a private API endpoint, it might not even be the same model the public gets. AA's countermeasure is a "mystery shopper" policy — registering accounts off its own domain and re-running benchmarks unidentifiably. Labs accept this because each wants assurance that competitors can't game the system either. That sentence alone tells you how adversarial evaluation has become.

What rigorous evaluation actually looks like#

The field isn't helpless. The serious evaluators have converged on a set of practices that make scores harder to game and easier to trust:

  • Held-out and private test sets. Questions never published can't leak into training data. Microsoft's MMLU-CF keeps its test set closed-source; private evals used by labs and independent auditors follow the same logic.
  • Dynamic benchmarks. Questions generated fresh at evaluation time from templates, so memorization has nothing to grab onto.
  • Contamination audits. N-gram overlap checks between training data and test items, paraphrase-invariance probes like the Stanford 2026 study's, and per-model contamination analyses should accompany every reported score. Reporting a benchmark without a contamination analysis is increasingly seen as incomplete.
  • Run evals yourself. Independent third parties (Artificial Analysis, Epoch AI, METR, and the UK/US AI Safety Institutes with frameworks like Inspect) re-run evaluations with identical harnesses, prompts, and repeat counts across all models — because lab numbers aren't comparable.
  • Report uncertainty. Repeats, confidence intervals, and variance estimates turn a point score into an honest measurement. The Stanford contamination study's recommendation — that leaderboards report paraphrase-controlled rankings alongside confidence intervals — is a good template.
  • Build your own eval. For anyone deploying a model, the canonical benchmark scores are nearly irrelevant. The only evaluation that predicts how a model will do on your task is an evaluation on your task, with your data. Everything else is a proxy.

The takeaway#

Benchmarks aren't useless — contamination inflates scores but mostly preserves rankings, and harder tests like Humanity's Last Exam and agentic benchmarks still discriminate. But the numbers in a launch deck are the beginning of an investigation, not its conclusion. They were produced on tests the model may have seen, using prompts tuned for the test, on benchmarks the field has already started to outgrow, with variance the table doesn't show.

Treat any benchmark score the way a scientist treats a single measurement: ask how it was taken, what could contaminate it, how much it varies, and whether it predicts anything you care about. The science of measuring AI is catching up with the science of building it — slowly, one closed test set at a time.