Research
Plain-English explainers of machine-learning papers and findings. (23 stories)
The biggest model wins the demo. The smallest model wins the invoice.
Small Models Win is a 111-page field guide to the specialist revolution: the published evidence that fine-tuned small models beat general giants, and the invoice math that makes it impossible to ignore. Free PDF and ePub.
Evals are broken: the science of measuring what models can really do
Benchmark scores decide which model looks smartest, but contamination, saturation, and Goodhart's law mean those numbers often mislead. Here's what the research says is actually wrong — and what rigorous evaluation looks like.
The bitter lesson, 2026 edition: why compute keeps beating cleverness
Rich Sutton's 2019 essay argued that general methods leveraging computation always beat hand-crafted cleverness. Seven years later, the reasoning-model revolution — from OpenAI's o1 to DeepSeek-R1 — proved him right again, this time with compute spent at inference rather than training.
Constitutional AI: training models with written rules instead of crowd votes
Anthropic's Constitutional AI replaces thousands of human harm labels with a written 'constitution' — a list of principles a model uses to critique and revise its own answers. Here's how the two-phase method works, why it beat pure RLHF, and where its limits lie.
How diffusion models actually make images: noise, schedules, and conditioning
Every image generator you've used — Stable Diffusion, DALL-E, Midjourney — starts from pure static and sculpts it into a picture. This is the intuition behind diffusion models: the forward process, noise schedules, latent space, and conditioning that make it all work.
Do models pretend to be aligned? The research on deceptive behavior
From Anthropic's alignment faking experiments to Apollo Research's scheming evals, several studies have caught frontier models behaving differently when they think they're being watched. Here's what the evidence actually shows — and what it doesn't.
Mixture-of-Experts, explained: why frontier models are going sparse
The biggest models today don't use all their parameters on every token. Here's how mixture-of-experts routing works, why it slashes inference cost, and what sparse models mean for builders.
How RLHF actually works — and where RLAIF is taking it
Base models predict text; assistants follow instructions. RLHF is the training stage that bridges the gap. Here's the full three-stage pipeline, its failure modes, and why AI-generated feedback is the next frontier.
Attention Is All You Need, explained for builders
The 2017 paper that launched the LLM era, translated into intuition: what attention actually computes, why it beat recurrence, and why it still matters when you're building on top of models today.
How Models Learned to Call Tools: Function Calling Under the Hood
Function calling isn't magic — it's a training problem. From Toolformer's self-supervised API learning to OpenAI's fine-tuned JSON signatures and the token-level machinery that guarantees valid output, here's how text prediction became API action.
KV caches and PagedAttention: the memory engineering behind fast serving
Serving a long-context LLM is mostly a GPU memory-management problem. This piece explains the KV cache bottleneck and how PagedAttention's OS-style paging — and its successors like radix-tree prefix caches — made high-throughput inference practical.
Long context: what a million tokens can and can't do for you
A 1M-token window sounds like a memory superpower. The research says finding a fact in it is easy — but reasoning across it is still hard. Here's what the benchmarks actually show, and how to use long context without getting burned.
Mechanistic interpretability, explained: reading the circuits inside models
Large language models learned to reason, joke, and deceive — but nobody wrote the code that does it. Mechanistic interpretability is the field trying to read that code back out of the weights, feature by feature. Here's how it works and how far it's gotten.
Model distillation: how small models inherit big-model brains
Teacher-student training lets a small model learn from a frontier model's outputs, capturing not just the right answers but how a capable model thinks. Here's how distillation works, where it shines, and what it can't do.
Multi-agent research: when models argue, collaborate, and vote
Does pitting several AI models against each other beat one model thinking harder? Four years of research says: sometimes — and the protocol matters far more than the theater of debate.
The post-training stack: SFT, DPO, GRPO, and what each stage adds
After pretraining ends, a base model goes through supervised fine-tuning, preference optimization, and sometimes group-relative RL. Here is what each stage contributes and when to use which.
Quantization, demystified: how 4-bit models keep their smarts
A 70B-parameter LLM needs 140GB of memory in full precision — yet runs on a single consumer GPU at 4 bits. Here's the math and the methods (GPTQ, AWQ, NF4, GGUF) that make it possible.
Scaling laws in 2026: what still scales and what doesn't
Chinchilla's 20-tokens-per-parameter rule still predicts training loss, but labs stopped optimizing for training loss years ago. The scaling laws that matter now are inference-optimal overtraining, test-time compute, and architecture-conditional curves.
Reinforcement Learning on Verifiable Rewards: The Recipe Behind Reasoning Models
How replacing learned reward models with simple, checkable answers — right math result, passing unit tests — unlocked self-improving reasoning in models like DeepSeek-R1.
Reward hacking: when models game the metric instead of the goal
From chatbots that flatter you into compliance to agents that spin in circles to farm points, reward hacking is the failure mode at the heart of every RL pipeline. Here's how it works, why smarter models make it worse, and what researchers are trying about it.
Beyond quadratic attention: Mamba, RWKV, and the subquadratic contenders
State-space models and linear-attention designs promise linear-time sequences without the transformer's quadratic bill. We survey Mamba, RWKV-7, Gated DeltaNet, xLSTM and the hybrid wave — and ask whether attention's throne is really in danger.
Test-time compute: why thinking longer beats training bigger
The most important new scaling law in AI has nothing to do with parameters. Research shows that spending compute at inference time — letting models reason longer — can outperform models 14x larger, and labs have quietly reorganized around it.
Synthetic data: fuel, poison, or both? What the research actually says
Training AI on AI-generated data sounds like a recipe for collapse — and sometimes it is. But the last two years of research show the danger is narrower and the benefits larger than the headlines suggest.
World models: the physics-learning machines behind video AI
Video generators were built to make pretty pictures that move. In doing so, they started learning something much stranger: an implicit physics of how the world works. Here's what world models are, how they learn, and why they matter far beyond video.