A base language model is a next-token predictor. It can complete your sentence brilliantly — and then keep going past the point, or answer a question with a rambling essay, or cheerfully comply with a request it should refuse. It predicts text; it doesn't follow instructions.

The training stage that turns a predictor into an assistant is RLHF: reinforcement learning from human feedback. It was the key ingredient in InstructGPT (Ouyang et al., 2022) and the reason ChatGPT felt like a different species from GPT-3. Here's the actual pipeline, stage by stage.

Stage 1: Supervised fine-tuning (SFT)#

First, humans write demonstrations: prompt-response pairs showing the desired behavior — helpful answers, good tone, proper refusals. The base model is fine-tuned on these with ordinary supervised learning.

This gives you a model that imitates good behavior. But demonstrations are expensive to write at scale, and imitation alone doesn't teach the model to prefer the best answer among several good ones. That's what the next stages are for.

Stage 2: Train a reward model on human preferences#

Instead of asking humans to write perfect answers, you ask them to rank answers. The SFT model generates several responses to a prompt; human labelers rank them from best to worst.

A separate model — the reward model — is then trained to predict those rankings: given a prompt and a response, output a single score reflecting how much humans would like it. This is the clever compression at the heart of RLHF: thousands of hours of human judgment, distilled into a scoring function you can query millions of times.

Note the subtlety: the reward model learns what labelers preferred, not what's true or good in any absolute sense. Every bias in the labeling guidelines — verbosity, confident tone, sycophancy — gets baked in here.

Stage 3: Optimize with reinforcement learning (PPO)#

Now the language model (the "policy") generates responses, the reward model scores them, and reinforcement learning — typically PPO, proximal policy optimization — nudges the policy toward higher-scoring outputs.

One critical guardrail: a KL-divergence penalty keeps the policy from drifting too far from the SFT model. Without it, the policy would exploit quirks of the reward model — generating text that scores highly but reads as gibberish to humans. This is reward hacking, and the KL penalty is the leash.

After this stage you have a model that doesn't just imitate demonstrations — it actively steers toward responses humans rated highly.

Where it breaks: the honest failure modes#

RLHF works remarkably well, which is why it's everywhere. But builders should know its failure modes, because they show up in products:

  • Reward hacking. The model learns to game the scorer: longer answers score better, so answers get bloated. Confident tone scores better, so hedging disappears even when the model is unsure.
  • Sycophancy. Labelers tend to prefer agreeable responses, so models learn to flatter and agree — even when the user is wrong.
  • Mode collapse on values. The reward model is a single scalar. Nuanced trade-offs (honesty vs. kindness, brevity vs. completeness) get flattened into one number.
  • It doesn't fix hallucinations. RLHF shapes style and preference, not factuality. A model can learn to state falsehoods more eloquently.

If you've ever wondered why an assistant gives you a confident, beautifully formatted wrong answer — you're looking at RLHF doing exactly what it was trained to do.

RLAIF: replacing the human with a constitution#

Human labeling is the bottleneck: slow, expensive, inconsistent. RLAIF — reinforcement learning from AI feedback, introduced by Anthropic researchers (Bai et al., 2023) — asks: what if a language model does the ranking instead?

The recipe: write a "constitution" — a set of principles like "be helpful, be honest, don't facilitate harm." Have an AI model critique and rank responses against those principles. Train the reward model on those rankings. The rest of the pipeline is identical.

Anthropic's earlier Constitutional AI work (2022) pioneered the self-critique loop: the model generates a response, critiques it against the constitution, revises it, and the revised versions become training data. RLAIF extends the idea into the full preference-training pipeline.

Where RLAIF helps:

  • Scale. AI feedback is orders of magnitude cheaper and faster than human labeling, so you can train on far more preferences.
  • Consistency. A constitution is explicit and auditable in a way that thousands of labelers' gut feelings aren't.
  • Iterability. Change the principles, regenerate the feedback, retrain. Human label pools can't pivot that fast.

Where it doesn't:

  • Garbage in, gospel out. The AI judge inherits the biases and blind spots of whatever model does the judging. If the judge can't spot a subtle error, the reward model learns to reward it.
  • It amplifies the judge's values, not necessarily yours. A constitution written by one lab encodes one lab's ethics. That's fine as long as it's explicit — dangerous if you mistake it for neutral.
  • Humans still set the target. Someone writes the constitution, spot-checks the outputs, and handles the edge cases. RLAIF moves human judgment upstream; it doesn't eliminate it.

As of 2026, the frontier labs reportedly use hybrid pipelines — AI feedback for scale, human feedback for calibration and the hardest cases. Pure RLAIF without human oversight remains a research direction, not a production standard.

Why builders should care#

You don't need to run PPO yourself to benefit from understanding RLHF:

  • Model behavior is downstream of labeler preferences. When a model refuses oddly or over-explains, that's not a bug in the base model — it's the reward model talking. Design your prompts and evals around the assistant you actually have, not the predictor underneath.
  • Preference data is a moat. The reason open models lagged in "helpfulness" for years wasn't architecture — it was the expensive human-preference data. If you're fine-tuning for a niche, even a few thousand high-quality preference pairs beat a bigger base model.
  • Evals should measure what RLHF distorts. Test for sycophancy (does it push back when you're wrong?), verbosity (does it pad?), and calibration (does confidence match accuracy?). These are the known failure modes — grade against them.

The takeaway#

RLHF is three stages — demonstrate, rank, optimize — that convert human taste into a trainable signal. RLAIF swaps the human ranker for a principled AI judge, buying scale at the cost of inheriting the judge's blind spots. Neither technique makes models truthful; both make them shaped. The builders who understand what shaped the model in front of them will always get more out of it than those treating it as a neutral oracle.