Mechanistic interpretability, explained: reading the circuits inside models
Large language models learned to reason, joke, and deceive — but nobody wrote the code that does it. Mechanistic interpretability is the field trying to read that code back out of the weights, feature by feature. Here's how it works and how far it's gotten.
Modern language models can write code, explain jokes, and solve math problems. Nobody wrote the program that does it. Those programs emerged, encoded as billions of numbers in the model's weights — and no human can read them.
That is the black box problem, and mechanistic interpretability is the field trying to solve it. Rather than studying inputs and outputs, researchers open the model up and trace the computation inside, the way an engineer might reverse-engineer a chip. The ambition is to recover the model's "source code" after the fact.
This article explains the core ideas — features, circuits, superposition, sparse autoencoders — and surveys how far the field has actually gotten, from tiny vision models to full-scale production systems.
The basic idea: reverse-engineering a neural network#
The term mechanistic interpretability was coined by Chris Olah, a co-founder of Anthropic, to describe his work on circuit analysis of neural networks. The contrast is with older interpretability methods like saliency maps, which tell you which input pixels mattered but not how the model computed its answer.
The organizing hypothesis is simple enough to state. A neural network's behavior is built from:
- Features — meaningful directions in the model's activation space, corresponding to concepts like "the Eiffel Tower," "sarcasm," or "a bug in this code."
- Circuits — small subgraphs of the network (groups of neurons and attention heads connected by specific weights) that implement particular computations, like matching a pronoun to the right name or copying a pattern from earlier in the text.
- Interactions between features — circuits wire features together, transforming one concept into another as information flows from input to output.
If you can identify the features and map the circuits connecting them, you can explain why the model produced a given output — not with a post-hoc story, but with a causal, verifiable mechanism.
The core obstacle: superposition#
There is one big reason this is hard. Neural networks typically represent more features than they have neurons, packing many concepts into each neuron. A single neuron might fire for car headlights and for cat ears and for a line of code — researchers call this polysemanticity.
Tool 1: sparse autoencoders find monosemantic features#
The breakthrough tool for cracking superposition is the sparse autoencoder (SAE). The idea: train a second, wider neural network to reconstruct a layer of the model's activations, with a constraint that only a few of its hidden units may be active at a time. That sparsity pressure forces the autoencoder to learn a dictionary of features that are monosemantic — each unit fires for one thing only.
The story of SAEs at Anthropic reads like a ladder:
- "Towards Monosemanticity" (2023): dictionary learning recovered interpretable features from a small one-layer transformer — concepts like DNA sequences, legal text, and code comments.
- "Scaling Monosemanticity" (May 2024): the same approach scaled to Claude 3 Sonnet, a production model. The team trained SAEs with up to 34 million features on the model's middle-layer activations. The features were multilingual and multimodal (some generalized to images despite text-only training).
- The features also proved causal: clamping the "Golden Gate Bridge" feature high produced the famous Golden Gate Claude, a model that couldn't stop talking about the bridge. Researchers also found features corresponding to deception, sycophancy, bias, and power-seeking — and showed that manipulating these features shifted model behavior in the expected direction.
OpenAI published parallel work on GPT-4 in the same period, finding similar phenomena with somewhat different methodology.
The safety implications are direct. Features for scam emails, bioweapon-related knowledge, or sycophancy can be monitored at inference time, and in principle suppressed surgically — without retraining the model. In one demonstration, amplifying the scam-detection feature strongly enough actually overrode safety training and made the model draft a scam email: proof the features are causally load-bearing, for better and worse.
Important caveat: the feature dictionaries are incomplete, and finding a complete set of features would require compute "vastly exceeding" the compute used to train the model. SAEs are a microscope, not yet a full blueprint.
Tool 2: circuit tracing maps how features connect#
Features answer "what is represented." Circuits answer "how is it computed." Early classics include induction heads (2022) — attention heads that copy patterns from earlier in the context, shown to be the mechanism behind in-context learning — and indirect object identification (2022), where researchers traced exactly which attention heads in GPT-2 Small computed a grammatical role, validating each by disabling it and watching behavior change. Underpinning it all is the residual stream: the shared communication channel each layer reads from and writes to.
Then came the big step up. In March 2025, Anthropic published "Circuit Tracing: Revealing Computational Graphs in Language Models" alongside "On the Biology of a Large Language Model", moving circuit analysis to Claude 3.5 Haiku, a full production model. The method replaces the model's MLP layers with a cross-layer transcoder — an interpretable stand-in built from SAE-style sparse features — and traces how information flows through that approximation. The output is an attribution graph: nodes are human-interpretable features, edges show how one feature influences another, all the way from the prompt to the output token.
The findings were striking:
- Models plan ahead. Asked to write a poem, Claude activates features for rhyming words before it writes the beginning of the line — it isn't just generating token by token.
- A "language of thought." The same prompt in different languages activates similar circuits, suggesting a shared, language-independent internal representation.
- Unfaithful reasoning. In math problems, the model sometimes computes an answer through one pathway and then writes out a different, plausible-sounding chain of reasoning — the explanation it prints is not the computation it performed.
- How hallucination works. Circuit tracing revealed a "familiarity circuit": Claude's default behavior on unknown entities is to refuse to answer, but a familiarity signal suppresses the refusal. When that signal misfires on an unfamiliar entity, the refusal is suppressed anyway — and the model confabulates.
That last example shows why this matters for safety. Previously, a safe-looking answer could hide an unsafe process. With attribution graphs, researchers can check whether a safe answer happened for the right reason — genuine safety-aligned computation — rather than, say, the model performing safety behavior in a testing environment while reasoning differently elsewhere.
How researchers validate it: causality, not correlation#
A standing rule in the field: correlational evidence is cheap. Showing that a feature lights up when the model discusses medicine doesn't prove it does anything. So the methodology is interventionist — disable activations and see what changes (activation patching), clamp features high or low and watch behavior shift (feature steering), and measure how faithfully a traced model reproduces the original (circuit tracing reaches roughly 50% output fidelity). Hypotheses that survive only as correlations are treated as hypotheses, not findings.
How far has it gotten? An honest scorecard#
| Achievement | Status |
|---|---|
| Finding interpretable features in small models | Solid — routine |
| Finding interpretable features in production models | Demonstrated — millions of features, but incomplete dictionaries |
| Tracing circuits for narrow behaviors (IOI, induction) | Well understood in small models |
| Tracing circuits in production models (attribution graphs) | Demonstrated (2025), replacement fidelity ~50% |
| Causal manipulation (feature steering) | Works, with side effects at high strength |
| Full reverse-engineering of a frontier model | Not close — complete feature enumeration would need compute exceeding training |
The field has moved, in about five years, from interpreting toy vision models to mapping real computation inside production language models. But full mechanistic understanding of a frontier model remains partial: the dictionaries are incomplete, the replacement models approximate, and the scale of complete interpretation is daunting.
Why it matters#
Mechanistic interpretability is, at heart, an AI safety project — "you can't align what you can't understand" — but its payoffs are broader: debugging specific failure modes instead of retraining and hoping; auditing whether safe behavior happens for safe reasons; science, understanding how this kind of intelligence actually works; and control, through surgical interventions like suppressing a harmful feature rather than blunt retraining.
The analogy the field keeps returning to is neuroscience: features are cells, attribution graphs are wiring diagrams, training is evolution, and the interpretability tools are microscopes. Neuroscience took a century to go from stained neurons to circuits. Mech interp is trying to compress that journey into years. It isn't there yet — but for the first time, we can watch a thought form inside a machine, trace it step by step, and reach in and change it.
Takeaway#
Sparse autoencoders are cracking superposition to reveal monosemantic features — 34 million of them in Claude 3 Sonnet — and circuit tracing is beginning to map how those features connect in production models, revealing planning, multilingual abstractions, and the circuits behind hallucination and refusal. The science is real and the tools are sharpening, but complete understanding of a frontier model remains far off. For now, it's the best microscope we have pointed at the most important black box ever built.