Here's a strange fact about modern frontier models: the biggest ones don't use most of themselves. A model advertised at 400 billion parameters might only activate 30 billion of them for any given token. The rest sit idle, waiting for tokens they're actually good at.

This is the mixture-of-experts (MoE) architecture, and it's quietly become the default way to scale language models. Here's how it works and why it matters to anyone building on these models.

Dense vs. sparse: the core trade-off#

A traditional "dense" model uses every parameter for every token. If you want a smarter model, you make it bigger — and every token gets more expensive, forever. Capability and inference cost are locked together.

MoE breaks the lock. Instead of one giant feed-forward network in each Transformer block, you have many smaller ones — the experts — plus a tiny router (or gating network) that decides, per token, which experts should handle it. Only the chosen experts run. The rest stay dark.

The result: you can scale total parameters (knowledge capacity) enormously while keeping active parameters per token (compute cost) roughly flat. Big brain, small electricity bill — relatively speaking.

How routing actually works#

For each token, at each MoE layer:

  1. The router — a small learned linear layer — scores every expert for that token.
  2. The top-k experts are selected (k is usually 1 or 2).
  3. The token is processed by those experts, and their outputs are combined, weighted by the router's scores.

A concrete example: Mixtral 8x7B (Mistral, December 2023) has 8 experts per MoE layer and routes each token to the top 2. Total parameters: ~47B. Active per token: ~13B. So it carries the knowledge capacity of a 47B model at roughly the inference cost of a 13B one.

The idea isn't new — the modern form traces to Google's Switch Transformer (2021), which showed that routing each token to just one expert (top-1) could scale to trillions of parameters. What changed is that the technique matured from research curiosity to production default.

Do experts specialize in interpretable topics — one for code, one for French? Early researchers hoped so. In practice, specialization is murky: experts develop statistical niches, but they rarely map to clean human categories. Think of it less like departments in a company and more like... habits the model fell into.

Why labs love it#

  • Capability per dollar. The binding constraint on frontier AI is training and inference cost. MoE buys more capability per FLOP than dense scaling — the single most important property in the industry right now.
  • Serving economics. Cheaper per-token inference means lower API prices and better margins. When you see a surprisingly capable model at a surprisingly low API price, MoE is often why.
  • Faster iteration. Smaller active compute per step means faster experimental turnaround during training.

The honest trade-offs#

MoE is not free lunch. Builders — especially those self-hosting open-weights models — should know the costs:

  • Memory, not compute, becomes the bottleneck. All experts must live in VRAM even though only a few run per token. Mixtral 8x7B needs ~100GB of VRAM in full precision — you pay memory for 47B parameters to get the compute of 13B. Quantization helps enormously here, which is why the open-weights MoE ecosystem runs on quantized builds.
  • Load balancing is fiddly. Left alone, routers develop favorites — a few experts get all the tokens, the rest atrophy. Training adds auxiliary losses to force even distribution. It's a solved problem but an inelegant one, and it interacts badly with some fine-tuning recipes.
  • Fine-tuning is harder. Updating an MoE model without wrecking the router's delicate balancing act takes more care than fine-tuning a dense model. Parameter-efficient methods (LoRA and friends) mostly work, but full fine-tunes need attention.
  • Latency variance. Different tokens take different expert paths, which can make batched inference less uniform. Serving systems have had to get clever about expert parallelism — splitting experts across GPUs.

What it means for builders#

If you use APIs: MoE is why frontier-class intelligence keeps getting cheaper per token. You don't need to think about it — but when comparing models, "total parameters" is now a vanity metric. Ask about active parameters per token; that's what determines speed and cost.

If you self-host: MoE models are the best capability-per-VRAM deal in open weights, provided you can fit all experts in memory. A quantized MoE often beats a dense model of the same memory footprint. Check VRAM requirements before falling in love with a parameter count.

If you fine-tune: prefer dense models for small custom fine-tunes (simpler, more predictable), and reach for MoE when you need maximum base capability — using LoRA-style methods that leave the router mostly alone.

If you evaluate models for procurement: ask vendors whether a model is dense or MoE, and what the active parameter count is. Two models with the same "total parameters" can differ 3x in inference cost and latency. The spec sheets that only quote totals are telling you the vanity number.

The takeaway#

Mixture-of-experts is the industry's answer to the cruelest equation in AI: smarter models cost more per token. By activating only the experts each token needs, MoE scales knowledge without proportionally scaling compute — at the price of memory hunger and training complexity. It's the reason the frontier keeps advancing while API prices keep falling, and understanding it is the difference between reading a spec sheet and actually understanding what you're buying.