Reinforcement Learning on Verifiable Rewards: The Recipe Behind Reasoning Models

In January 2025, DeepSeek showed that a language model could teach itself to reason — reflecting, double-checking, and changing strategy mid-thought — using nothing but reinforcement learning and a reward signal that any script could compute. No human-labeled reasoning traces, no learned reward model. The technique now has a name: reinforcement learning with verifiable rewards (RLVR). It is arguably the single most important post-training idea behind the current generation of reasoning models, and understanding it explains why models suddenly got dramatically better at math, code, and other hard problems.

The problem RLVR solves: reward hacking#

Most people know the older pipeline: supervised fine-tuning (SFT), then RLHF, where a learned reward model trained on human preferences scores the model's outputs. That learned reward model is a soft target. In large-scale training, the policy finds inputs the reward model scores highly that humans would not — the classic failure mode called reward hacking (or over-optimization).

RLVR replaces the learned judge with a deterministic one. If the task has an objectively checkable answer, the reward is a rule: did the final math answer match the gold solution? Did the generated code pass its unit tests? Then 1, else 0. There is no neural surface to exploit — a unit-test pass has no "almost" that a model can slide into. The Allen Institute team that coined the term in the Tülu 3 report put it plainly: RLVR only grants reward when the model's completions are verified to be correct.

The price of that robustness is scope. RLVR works only where correctness can be checked mechanically, which is why it broke through first in math, code, and logic. Open-ended writing still needs learned rewards or AI judges. But the surprise of the last two years is how far the verifiable recipe reaches once it escapes its original habitat.

The three kinds of verifiable rewards#

Following the taxonomy from the RLVR literature, there are three flavors:

DeepSeek-R1 used accuracy rewards (did the answer match?) plus a small format reward (did the model put its reasoning inside the designated tags?), and deliberately avoided any neural reward model.

GRPO: the cheap RL that made it affordable#

RLVR almost always rides on one algorithm: Group Relative Policy Optimization (GRPO), introduced by the DeepSeek team in the DeepSeekMath work (Shao et al., 2024). Standard PPO needs a separate critic network that learns to predict value — expensive at LLM scale. GRPO deletes the critic. Instead, for each prompt the model samples a group of candidate answers, scores each with the verifier, and computes each answer's advantage relative to the group's average. Answers that beat their siblings get reinforced; losers get suppressed.

A formal analysis of GRPO under verifiable rewards (Mroueh et al., 2025) showed this amounts to a contrastive loss between 0-reward and 1-reward samples, with a neat calibration property: rare successes get disproportionate credit, and wrong answers are penalized hardest exactly when success should have been easy. The original recipe combines PPO-style clipping with an explicit KL regularizer keeping the policy near a frozen reference model — cheap RL that made large-scale reasoning training affordable.

The R1-Zero experiment: reasoning from reward alone#

The landmark result is DeepSeek-R1-Zero. The team took DeepSeek-V3-Base and applied pure RL — no supervised fine-tuning first — with GRPO and rule-based accuracy and format rewards on math and reasoning tasks. What happened:

• Reasoning behaviors emerged on their own. Over thousands of RL steps, the model developed self-verification, reflection, and lengthening chains of thought. The "aha moment" — the model learning to re-examine its own work mid-generation — appeared spontaneously.
• Benchmarks jumped. AIME 2024 pass@1 rose from 15.6% to 71.0%, and with majority voting reached 86.7%, matching OpenAI's o1-0912 at the time.
• The catch: R1-Zero's outputs were often hard to read, with mixed languages and no clean formatting — which is exactly what you'd expect when the only goal is "get the checkable answer right."

The full DeepSeek-R1 pipeline fixed this with a four-stage process: a small cold-start SFT phase on curated long chain-of-thought data for readability, then reasoning-oriented RL, then rejection sampling and a second SFT, then a final RL pass — plus distillation that transferred the reasoning patterns into small dense models from 1.5B to 70B parameters. (The paper, arXiv:2501.12948, was later published in Nature in 2025.)

Tülu 3, the open project that gave RLVR its name, used the technique differently: as the final post-training stage after SFT and DPO, targeted at skills with verifiable answers like math and precise instruction following — and beat the instruct versions of Llama 3.1, Qwen 2.5, and Mistral at the same sizes.

The results nobody predicted#

Once the recipe was public, researchers stress-tested it and found behaviors that complicate the simple story:

• Spurious rewards still work — sometimes. A 2025 study (Shao et al.) ran RLVR on Qwen2.5-Math-7B with rewards that had little or no relation to the correct answer — random rewards, format-only rewards, even rewards for incorrect labels — and still gained 13.8–24.1 absolute points on MATH-500, nearly matching the 29.1 points from ground-truth rewards. The hypothesis: with no useful signal, the RL dynamics surface reasoning representations already learned in pretraining. The catch is that the effect is model-dependent — it worked for Qwen but largely failed on Llama 3 or OLMo 2, and code-flavored internal reasoning spiked from 65% to over 90% of generations regardless of reward.
• One example can be enough. A separate line of work showed that RLVR on a single training example lifted Qwen2.5-Math-1.5B from 36.0% to 73.6% on MATH-500, matching the result of training on a 1,200-example subset — with gains generalizing across benchmarks even after training accuracy saturated. Adding entropy loss alone, with no outcome reward at all, still added 27.4 points. RLVR looks less like it teaches new knowledge and more like it amplifies and organizes what pretraining already built.

Both findings point to the same lesson: verifiable RL is as much a capability elicitation technique as a teaching one. The reasoning latent in a strong base model can be drawn out by the pressure of verifiable success.

Limits and open questions#

RLVR is not a complete theory of reasoning models, and honest accounts flag the gaps:

• Domain-bound. Anything without a mechanical checker — nuance, taste, open-ended advice — still needs learned rewards or judges.
• Not unhackable. Right answers can be reached through unfaithful steps, and checkers can be gamed; weak KL regularization has been shown to let reward hacking creep back even with verifiable constraints. Faithfulness and response length still need watching.
• Model dependence. The spurious-reward and one-shot results hold strongly for some model families and weakly for others — a warning against treating any single leaderboard number as the recipe's verdict.

Takeaway#

The reasoning-model leap came from a deceptively simple insight: where you can check the answer, don't learn a judge — be the judge. Rule-based rewards on math and code, optimized with critic-free GRPO, proved strong enough that reasoning behaviors emerged from pure RL, scaling from DeepSeek-R1-Zero to an open ecosystem of RLVR-trained models. The frontier now is extending the recipe — model-based verifiers, harder checkable domains, and keeping the training honest — but the core idea will stay: a reward you can compute beats a reward you have to learn.