The post-training stack: SFT, DPO, GRPO, and what each stage adds
After pretraining ends, a base model goes through supervised fine-tuning, preference optimization, and sometimes group-relative RL. Here is what each stage contributes and when to use which.
A pretrained language model is a next-token prediction engine, not an assistant. It has read the internet and can complete text fluently, but it does not know that it should answer questions directly, refuse harmful requests, or format code blocks properly. Everything between the end of pretraining and the model you actually chat with is post-training — a stack of stages, each fixing a different gap.
The standard pipeline used by OpenAI (InstructGPT), Anthropic, Meta, and the open-source ecosystem is: SFT → preference modeling → policy optimization. Each stage answers a distinct question: how should it behave, which behaviors are better, and how do we push it there. Newer methods like DPO and GRPO are not new goals — they are cheaper, more stable ways to execute those stages.
Stage 1: SFT — teaching the assistant format#
Supervised fine-tuning (SFT), also called instruction tuning, is the first and simplest stage. You collect a dataset of (prompt, ideal-response) pairs — human-written or curated demonstrations of good assistant behavior — and train the model on them with the same next-token prediction objective as pretraining.
What SFT adds is the behavioral format: answering directly instead of continuing the prompt, following instructions, adopting a consistent tone, structuring responses. It is ordinary supervised learning on a deliberately narrow dataset — typically tens of thousands to hundreds of thousands of high-quality examples, tiny compared to pretraining corpora.
Its limitation is fundamental: SFT teaches imitation, and its ceiling is the quality of the demonstrations. The model can only copy strategies present in the data. It also suffers from distribution shift — at inference time the model conditions on its own previous tokens, which SFT never trained it to do. And SFT alone cannot express fine-grained trade-offs ("be helpful but refuse this kind of request") because its signal is binary: this response is the target, everything else is not.
Still, SFT is the foundation. In RLHF, the SFT model becomes both the starting policy and the reference point that later stages are constrained not to drift too far from. Even DeepSeek-R1, famous for pure RL, used a cold-start SFT phase to stabilize readability before RL took over.
Stage 2 (conceptually): preference data — from demonstrations to comparisons#
The key insight of RLHF is that demonstrations are expensive and shallow. It is much easier to get humans (or a strong model judge) to compare two outputs and say which is better than to write a perfect response from scratch. This produces preference datasets: (prompt, chosen, rejected) triples.
From these comparisons, the classic pipeline trains a reward model: typically the SFT model with its language-modeling head swapped for a scalar regression head, trained with the Bradley–Terry loss so that it scores the chosen response higher than the rejected one. The reward model is a proxy for "how much would a human like this response."
That is conceptually one stage — data collection plus reward modeling — but in the modern stack it is often merged with the next one.
Stage 3a: DPO — preferences without the RL loop#
Direct Preference Optimization (Rafailov et al., 2023, Stanford; NeurIPS 2023, arXiv:2305.18290) asked a striking question: if the reward model is learned from preferences and the policy is optimized against that reward under a KL constraint to a reference model, why learn the reward model at all?
The math turns out to be collapsible. The optimal policy for a KL-constrained reward objective has a closed form, and when you solve for the reward and substitute it into the Bradley–Terry preference model, the partition function cancels. What remains is a single classification loss computed directly on preference pairs: raise the log-likelihood of the chosen response, lower that of the rejected one, measured relative to a frozen reference model (usually the SFT checkpoint), with a temperature parameter β controlling how far the policy may deviate.
What DPO adds:
- No reward model, no RL loop. Training is fully differentiable supervised learning — no sampling from the policy, no PPO, no actor–critic instability, no separate value network.
- Stability and simplicity. The paper reported results competitive with or better than PPO-based RLHF on sentiment control, summarization, and dialogue with little hyperparameter tuning, using ordinary language-model training infrastructure.
- Accessibility. DPO meaningfully lowered the barrier to alignment: open models like Llama 2-Chat, Zephyr, and StableLM-Tuned could be preference-aligned without complex RL infrastructure.
The cost: DPO is offline — it trains on a fixed preference dataset rather than exploring. It cannot improve beyond what the data's comparisons imply, and it is sensitive to the cleanliness and diversity of the preference pairs. It spawned a family of variants (IPO, ORPO, SimPO), but the core idea — the language model's log-ratio against a reference is itself an implicit reward — remains a cornerstone.
Stage 3b: GRPO — RL without the critic, rewards without humans#
Group Relative Policy Optimization (Shao et al., DeepSeek-AI, introduced in the DeepSeekMath paper, arXiv:2402.03300, Feb 2024) attacked the other side of the pipeline. PPO needs a critic (value network) to estimate advantages, which is memory-hungry and hard to train well on long chains of thought. GRPO drops the critic entirely.
Instead, for each prompt, GRPO samples a group of completions (commonly 16–64) from the current policy, scores them, and computes each completion's advantage as its deviation from the group's mean reward, normalized by the group's standard deviation. Same prompt, same difficulty — the group mean automatically subtracts the prompt-specific baseline. The policy is then updated with a PPO-style clipped objective plus KL regularization against a reference model.
GRPO's most important pairing is RLVR — reinforcement learning with verifiable rewards. Instead of a learned reward model (human preferences), the reward comes from a rule-based checker: for math, the final answer is right or wrong; for code, the tests pass or fail. Cheap, exact, unhackable.
That combination is what powered DeepSeek-R1 (Jan 2025): pure GRPO on rule-based rewards, applied directly to a base model in the R1-Zero variant, spontaneously produced long chain-of-thought reasoning, self-reflection, and backtracking — behaviors that emerged from the reward signal rather than imitation. The reported figures: R1-Zero lifted AIME 2024 pass@1 from 15.6% to 71.0% (86.7% with majority voting), and the full DeepSeek-R1 reached 79.8% pass@1 on AIME 2024 and 97.3% on MATH-500, broadly matching OpenAI o1 on the paper's reasoning benchmarks.
What GRPO adds: the ability to go beyond the data. Unlike SFT (imitate) and DPO (rank), GRPO with verifiable rewards lets the model explore and discover strategies no human demonstrated. The trade-off is that it needs tasks with checkable answers — it does not work for open-ended style or taste.
The stack at a glance#
| Stage | Signal | Data needed | Adds |
|---|---|---|---|
| SFT | Demonstrations | (prompt, ideal response) pairs | Assistant format, instruction following |
| Reward model (+ PPO) | Human preferences | (prompt, chosen, rejected) triples + 3-model RL infrastructure | Alignment to taste, safety, helpfulness |
| DPO | Human preferences | Same preference triples, no reward model or RL | Same alignment goal, cheaper and more stable |
| GRPO + RLVR | Verifiable correctness | Prompts with checkable answers | Emergent reasoning, super-human solution strategies |
In practice, modern frontier pipelines run SFT → DPO-style preference alignment → GRPO-style reasoning RL in sequence. The R1 recipe is representative: a cold-start SFT, then large-scale GRPO on verifiable tasks, then rejection-sampled SFT on the RL checkpoint's own outputs (roughly 600k reasoning examples) mixed with non-reasoning data, and distillation into smaller dense models. Note the loop: RL creates the capabilities, SFT redistributes them.
Takeaway: a builder's map#
If you are post-training a model in 2026, the decision tree is simple:
- Start with SFT always. It is the cheapest stage and everything else assumes a model that speaks in assistant format.
- Use DPO (or a variant) for taste and safety. When the target is "which of these two answers would a user prefer," offline preference optimization is the stable, well-understood choice.
- Use GRPO + verifiable rewards for reasoning. When the target is correctness on math, code, or logic, rewarding checkable outcomes with group-relative advantages can discover capabilities no dataset contains.
- Remember the ceiling. SFT is capped by demonstrations, DPO by preference data, GRPO by what a reward checker can verify. The frontier is pushing verification further out — better automated checkers, model-based judges for open-ended tasks — because the method that can reward it can learn it.
Post-training is where raw capability becomes a product. Pretraining builds the engine; SFT teaches it the job description; preference optimization sets its values; and RL with verifiable rewards, at the frontier, teaches it to think.