Frontier AI models keep getting bigger, but the models people actually deploy keep getting smaller. The bridge between those two facts is distillation: a training technique where a large "teacher" model transfers its knowledge to a smaller "student" model, so the student can run on a laptop, a phone, or cheap cloud hardware while retaining much of the teacher's capability. Here's how the trick works.

The core idea: learn from the probabilities, not just the answers#

When you train a model the ordinary way, each training example gives it a hard label — this picture is a cat, that sentence is positive. The label says "cat: 100%, everything else: 0%," which tells the model what the right answer is but nothing about why.

Distillation, formalized by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in their 2015 paper "Distilling the Knowledge in a Neural Network" (arXiv:1503.02531), changes what the student learns from. Instead of hard labels, the student trains on the teacher's soft targets: the full probability distribution the teacher produces. If a teacher says "cat: 80%, dog: 15%, chair: 0.1%," that distribution carries what Hinton called "dark knowledge" — the information that cats resemble dogs far more than chairs.

That distinction matters. A student trained only on hard labels learns to memorize correct answers. A student trained on the teacher's soft outputs learns how the teacher generalizes — the teacher's uncertainty, its sense of similarity between classes, and which distinctions are hard. Sanh and colleagues showed the difference concretely with DistilBERT in 2019: a student trained on distilled soft supervision reaches performance you cannot get by training the same small architecture on hard-labeled data alone.

The mechanics: temperature and a combined loss#

The classic recipe has two ingredients:

  1. Temperature-scaled softmax. Normally a model outputs sharp probabilities (one class dominates). Raising the "temperature" parameter T of the final softmax spreads that distribution out, revealing the teacher's view of the also-rans. The student is trained at the same high temperature to match these softened targets.
  1. A hybrid loss. The student minimizes a weighted combination of (a) the divergence between its softened outputs and the teacher's softened outputs, and (b) the standard loss against the true hard labels (when labeled data exists). In practice for LLMs this is the KL divergence between the two temperature-scaled distributions, added to ordinary cross-entropy on ground truth.

There are several flavors of distillation. Response-based distillation matches final outputs — the standard approach for LLMs, done per token. Feature-based distillation goes deeper, matching the student's intermediate hidden states or attention patterns to the teacher's; it can transfer more nuanced behavior but costs more to compute and requires both models' internals. Relation-based distillation matches relationships between examples rather than outputs directly. In the LLM world these are usually discussed as white-box distillation (you have access to the teacher's logits, hidden states, or weights) versus black-box distillation (you only have API-level access to the teacher's inputs and outputs). Richer internal access generally transfers behavior more efficiently, but text-only black-box distillation is how many practical students are built — generate lots of high-quality text with a frontier model, then fine-tune a smaller open model on it.

Why distillation became the engine of the small-model boom#

The pre-training runs that produce frontier models are enormously expensive. Distillation amortizes that investment: pay once for the teacher, then create many cheap students.

The most influential example is DistilBERT (Sanh et al., 2019, Hugging Face). The team distilled BERT-base — 12 layers, 110M parameters — into a 6-layer, 66M-parameter student. The result retained 97% of BERT's performance on GLUE while being 60% faster at inference. That became the canonical proof that small models didn't have to be dumb models, and the "Distil-" naming pattern spread everywhere.

The same playbook scaled up with instruction-tuned chat models. Vicuna, released by LMSYS, was fine-tuned from LLaMA on conversation data collected from ChatGPT (a textbook case of black-box distillation from a closed teacher); LMSYS reported that GPT-4-based evaluation rated Vicuna's responses at about 90% of ChatGPT's quality — a striking claim that established distillation as a practical route to near-frontier chatbots. Around the same period, teams discovered that distilling reasoning traces — the teacher's chain-of-thought, not just its final answers — transfers how to reason step by step, a line of work including "Distilling Step-by-Step" (Hsieh et al., 2023), which used rationales to train much smaller students.

The flagship recent example is DeepSeek-R1 (January 2025). DeepSeek trained a 671B-parameter reasoning model with large-scale reinforcement learning, then distilled its behavior into six open dense models ranging from 1.5B to 70B parameters, based on Qwen and Llama architectures, using a reported 800,000 supervised samples of R1's reasoning traces. DeepSeek reported that the R1-Distill-Qwen-32B model outperformed OpenAI's o1-mini on several benchmarks. Distilling worked better than trying to run the same RL directly on the smaller models — a lesson that now shapes how reasoning capabilities get distributed: frontier RL happens once at scale, and SFT-based distillation spreads the result.

What distillation can and cannot copy#

Distillation is powerful but bounded, and it's worth being clear about the limits:

  • Capacity is a ceiling. A student can't reliably learn behaviors that require more representational capacity than it has. Distillation narrows the gap between teacher and same-sized-from-scratch training; it doesn't eliminate the gap between 1.5B and 671B parameters.
  • Errors transfer too. The student inherits the teacher's blind spots, biases, and hallucinations — dark knowledge includes dark ignorance. If the teacher is confidently wrong, the soft targets teach the student to be confidently wrong in the same way.
  • Novel data still matters. Distillation is strongest at transferring capability on distributions the teacher already handles. A student that must operate in a new domain still needs real data or further training there.
  • It's not the same as compression. Quantization and pruning shrink an existing model; distillation trains a new, smaller model from the teacher's outputs. The techniques combine well — the DeepSeek distills, for instance, are routinely run quantized — but they solve different problems.

The takeaway#

Distillation reframes what "training a model" means. The scarce, expensive input is the frontier teacher's intelligence — learned from web-scale data and massive compute. Once that exists, students can be manufactured comparatively cheaply by anyone with access to the teacher's outputs and modest training budgets. That's why the practical LLM landscape is increasingly teacher-at-the-top, students-everywhere: a few enormous models set the capability frontier, and distilled versions of their behavior run on laptops, phones, and $2/hour GPUs. The technique is a decade old, but its role as the distribution mechanism for AI capability is only growing — and the legal and provenance questions around distilling from closed teachers are far from settled.