Test-time compute: why thinking longer beats training bigger
The most important new scaling law in AI has nothing to do with parameters. Research shows that spending compute at inference time — letting models reason longer — can outperform models 14x larger, and labs have quietly reorganized around it.
For four years, progress in AI had one formula: more parameters, more data, more training compute. Bigger was better, predictably. Then, in late 2024, the field quietly found a second axis of scaling — one that costs nothing to train and everything to think. It's called test-time compute scaling, and it may matter more than the first one.
The premise is simple: instead of answering instantly, a model spends more computation at inference time — generating longer chains of thought, exploring multiple reasoning paths, checking its own work — and gets dramatically better results. The research says that for hard reasoning problems, a smaller model allowed to think long enough can beat a model many times its size that answers in one shot.
From "train bigger" to "think longer"#
The original scaling laws (Kaplan et al., 2020; Chinchilla, 2022) described training: loss falls predictably as you scale parameters, data, and training FLOPs. They never said much about what happens after training, because for years inference was just a thin wrapper around a single forward pass.
That changed in September 2024, when OpenAI released o1. Rather than scaling parameters, o1 used large-scale reinforcement learning to train models to produce extended internal chains of thought before answering. The model generates thousands of hidden "reasoning tokens" — exploring strategies, catching its own errors, refining its approach — and only then delivers a final answer. On GPQA Diamond (PhD-level science questions), GPT-4o scored 53.6%, o1-mini scored 60.0%, and o1-preview reached 73.3%, per OpenAI's launch report.
The more surprising finding from OpenAI's report: performance scaled smoothly with test-time compute itself. More thinking budget, better results — a new scaling curve that hadn't plateaued. Suddenly labs didn't only need bigger models; they could get capability by letting existing models think longer.
The paper that formalized it: Snell et al. (2024)#
Berkeley researchers Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar turned this intuition into a systematic research program in "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters" (arXiv 2408.03314, August 2024).
Their core question: given a fixed total budget of training plus inference compute, where should you spend it? They studied two mechanisms for scaling test-time computation:
- Search against verifiers — generating multiple candidate responses and scoring them with reward models, including dense process-based reward models that evaluate intermediate reasoning steps.
- Adaptive updating — modifying the model's output distribution at test time based on the prompt itself.
Two findings made the paper:
- A compute-optimal allocation strategy beat plain best-of-N sampling by more than 4× in efficiency. How you spend inference compute matters as much as how much you spend: the optimal mix of strategies depends on the difficulty of the specific prompt, so allocating compute adaptively per-prompt dominates any fixed recipe.
- A smaller base model with scaled test-time compute outperformed a 14× larger model in a FLOPs-matched comparison — but only on problems where the smaller model already achieved non-trivial success rates. Test-time compute amplifies capability; it can't conjure it from nothing.
That last caveat is load-bearing, and we'll come back to it.
The three ways to spend inference compute#
In practice, test-time compute comes in three flavors, and the research shows each has a different regime where it shines:
| Strategy | What it does | Best for |
|---|---|---|
| Parallel scaling (best-of-N, self-consistency) | Generate N independent candidates; pick the best via a verifier or majority vote | Easy-to-medium problems with reliable verifiers |
| Sequential scaling (long chains of thought, revision) | One long reasoning trace with self-correction and verification steps | Medium problems where the model can check its own work |
| Search (beam search with process reward models) | Explore and score a tree of reasoning steps, pruning bad branches | Hard structured problems — math, code, logic |
The process reward model (PRM) deserves emphasis. OpenAI's "Let's Verify Step by Step" showed that rewarding each reasoning step rather than just the final answer (outcome reward) produces better verification — and better verification makes all search strategies more effective. Hybrid strategies combining parallel sampling, PRM-guided search, and sequential revision were the most efficient in Snell et al.'s experiments.
The industry pivoted hard#
OpenAI's o1 was the proof of concept; the open-source world provided the replication. In January 2025, DeepSeek released DeepSeek-R1, showing that strong reasoning capabilities could be incentivized through pure reinforcement learning — no human-labeled reasoning demonstrations required. Their R1-Zero variant developed reasoning patterns like self-reflection and verification spontaneously during RL (the famous "aha moment"), and the final model matched or exceeded conventionally trained models on verifiable tasks in math, coding, and STEM. Crucially, those reasoning patterns could then be distilled into much smaller models, compressing the capability into deployment-friendly sizes.
By 2025, the shift was industry-wide: Anthropic shipped extended-thinking modes in Claude, Google built reasoning into Gemini, and OpenAI's o3 pushed the test-time budget further still. Meanwhile, providers exposed the trade-off directly to users: APIs now offer reasoning_effort or thinking-budget parameters, letting you dial quality against cost per query — a literal pay-per-thought pricing model.
Where the law holds — and where it breaks#
Test-time compute scaling is a real, measured phenomenon, but it is not a universal law. The research is clear about the boundaries:
- It needs verifiable structure. Math, code, logic puzzles, and science questions — tasks where correctness can be checked — scale beautifully. For open-ended creative work or tasks requiring world knowledge, more thinking doesn't reliably help, because there's no reliable signal to search against.
- It saturates. Accuracy improves roughly log-linearly with thinking budget, meaning each doubling of compute buys a similar increment of quality — expensive increments. Diminishing returns are always waiting.
- The base model matters. Snell et al.'s 14×-model finding held only where the small model was already non-trivially competent. Inference compute is an amplifier, not a miracle.
- It shifts the cost model, not the cost. A reasoning model can generate 10–100× more tokens than a standard model on a single query (10K+ thinking tokens versus a few hundred). Hidden reasoning tokens mean higher per-query cost and latency — 10 to 60 seconds of "thinking" on hard problems. The question "should we scale inference?" is really "is this query worth the extra cents?"
That cost framing is why this is arguably an economics story as much as a research one. Training compute is a one-time capital expense; inference compute is a recurring operating expense. As models saturate on training scaling, value migrates to whoever runs inference most efficiently — and smaller models with big thinking budgets (think o1-mini outscoring GPT-4o) become the production sweet spot: lower memory, lower serving cost, competitive quality.
The takeaway#
The field spent years asking "how big should the model be?" Chinchilla answered that for training. Test-time compute asks the mirror question: "how hard should the model think?" — and the early research suggests the answer, for reasoning tasks, is harder than we'd guessed.
For builders, the practical implications are already here: expose thinking budgets, match strategy to verifier quality, and don't assume a 14×-larger model beats a smaller one that's allowed to reason. For everyone else, the lesson is simpler and stranger: the frontier of AI capability is no longer just the models we train — it's the patience we give them when we ask.
Sources: Snell et al., "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters" (arXiv:2408.03314); DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (arXiv:2501.12948); OpenAI, "Learning to Reason with LLMs" (September 2024); Towards AI analysis of the o1 report's test-time scaling laws; survey chapter on test-time compute scaling.