Every token the transformer reads has to shake hands with every other token. That's the quadratic bill at the heart of attention: double the context, quadruple the compute — and watch the KV cache balloon. For years, engineers worked around it with sparse patterns, FlashAttention kernels, and KV-cache quantization. But a different camp asks a more radical question: what if we stop paying attention altogether?

The contenders go by many names — state-space models (SSMs), linear attention, gated recurrences — but they share one pitch: sequences processed in near-linear time, with inference memory that stays roughly constant instead of growing with context. A few years ago they were curiosities. In 2026, they are powering shipping frontier models. Here's where the race actually stands.

The quadratic bill#

In standard softmax attention, a sequence of length N produces an N×N attention matrix. Memory and compute grow with N², which is why serving a million-token context is a heroic feat of engineering rather than a casual option. The KV cache — the stored keys and values of every past token — grows linearly with N and is the main reason long-context inference needs warehouse-scale GPUs.

The subquadratic alternatives all attack this differently, but the intuition rhymes: keep a compressed "state" that summarizes the past, update it as each new token arrives, and generate from it. That gives O(N) training compute and constant memory per generated token — exactly the property that made the original RNN family attractive, minus the old RNN's vanishing-gradient and training-parallelization problems.

Mamba: the selective state space#

The modern wave started with Mamba (Gu & Dao, December 2023). Classical SSMs like S4 map sequences through a latent state using fixed transition matrices — a linear time-invariant system that can be computed as a global convolution. Mamba's twist was selectivity: the SSM's parameters (the B, C, and Δ matrices in the state update) become functions of the input itself. The model learns to filter — amplifying important tokens, suppressing irrelevant ones — which is what gives attention its expressive power.

Two engineering details made it work in practice: a hardware-aware parallel scan algorithm (recurrence during training would otherwise serialize everything), and a stripped-down block design that drops both the attention module and the MLP. The headline result: linear-time complexity, constant memory during generation, and performance that matched transformers on language benchmarks — at a reported 5× inference speedup on long sequences versus transformers in the original paper.

Mamba-2 (2024) generalized the idea further with "state space duality," reframing SSMs as a structured attention variant and scaling the architecture up with larger states and heads. It became a popular ingredient in hybrid designs.

RWKV: the RNN that trains like a transformer#

RWKV (pronounced "RWaKuV") takes the most deliberately heretical position: it's a pure recurrent neural network with no attention mechanism at all, yet it trains in parallel like a transformer. The trick is a time-mixing mechanism whose recurrence can be reorganized into a form that parallelizes across the sequence during training, then collapses back to constant-memory, constant-time-per-token inference.

The newest generation, RWKV-7 "Goose" (paper released March 18, 2025), introduces what the authors call dynamic state evolution: a generalized delta rule with vector-valued (channel-wise) gating and in-context learning rates. The interesting part isn't just the engineering — it's a provable expressivity result. Under standard complexity conjectures, transformers are limited to the circuit class TC⁰. RWKV-7 provably recognizes all regular languages and performs state tracking, placing it beyond that boundary while keeping training parallelizable.

The released RWKV-7 models (0.19B to 2.9B parameters, trained on a 3.1-trillion-token multilingual corpus, Apache 2.0) established a new state of the art at the 3B scale on multilingual tasks and matched English-language peers — trained on dramatically fewer tokens than competing 3B models, according to the paper.

Linear attention: maybe the softmax is the problem#

A third family keeps the attention skeleton but asks whether softmax is the actual bottleneck. Replace the softmax kernel with a linear one, and attention rearranges into a recurrence: O(N) time, constant-size state, no KV cache.

The standout here is Gated DeltaNet (ICLR 2025). It combines the delta rule — a memory update borrowed from old fast-weight neural network research that lets the model overwrite specific memories rather than just decay them — with Mamba-style gating. The result consistently beat Mamba-2 across language modeling, reasoning, and retrieval benchmarks in its evaluations, and it has become the de facto linear-attention layer of choice: it powers Alibaba's Qwen3-Next (February 2026, 80B total parameters with only 3B active per token) and appears in several other hybrid models.

Two neighboring ideas round out the field. Hyena (Poli et al., 2023) is a drop-in attention replacement built from implicit long convolutions alternated with data-controlled gating, keeping the data-dependent expressiveness of attention at subquadratic cost. xLSTM (NeurIPS 2024 spotlight, from Sepp Hochreiter — the co-inventor of the original LSTM) modernizes LSTMs with exponential gating and two block types (sLSTM with scalar memory, mLSTM with fully parallelizable matrix memory), and xLSTM 7B emerged as one of the most efficient 7B models around.

The real winner so far: hybrids#

Here's the honest scorecard. Pure SSM and linear-attention models are remarkably strong — but the flagship models shipping to users in 2026 are hybrids, keeping a minority of full-attention layers alongside the subquadratic machinery.

The template was set by Jamba (AI21 Labs, March 2024): a 52B-parameter model (12B active) interleaving transformer and Mamba layers at a 1:7 ratio, plus mixture-of-experts. The ablation is telling — pure Mamba lagged on format adherence and in-context learning tasks like IMDB and QuAC, and even a small dose of attention restored transformer-like behavior. The payoff: Jamba matched Mixtral-8x7B-class quality with only 4GB of KV cache at 256K context (versus 32GB for Mixtral) and up to 3× the throughput.

Qwen3-Next follows the same playbook at larger scale: 48 layers, 36 of them Gated DeltaNet, every fourth layer full attention, and an ultra-sparse MoE (512 experts, 3B of 80B active). It claims 10× higher throughput than its dense predecessor on contexts beyond 32K tokens. RWKV-X (April 2025) is the mirror image from the other direction: RWKV-7 blocks fused with top-k chunk sparse attention, near-perfect 64K passkey retrieval.

The pattern is consistent enough to call it a law of the field: the subquadratic layers do the bulk carrying of long context cheaply, and a thin scaffolding of full attention preserves the recall and in-context learning that softmax attention does best.

How close are they to dethroning the transformer?#

Closer than ever, but "dethrone" is the wrong verb. Consider three angles:

  • Efficiency: Decisive win. Linear-time training and constant-memory decoding are architectural facts, not benchmarks. The hybrid flagships demonstrate it at scale — 256K contexts, 10× long-context throughput, tiny KV caches. On long-sequence workloads, the transformer as we knew it has already lost.
  • Quality at frontier scale: Still contested. The largest, most capable models training today remain attention-dominant. SSMs and linear attention have closed the gap impressively up to tens of billions of parameters, but nobody has yet trained a pure subquadratic model that unambiguously beats a transformer at the frontier — and frontier labs keep their architecture bets close.
  • Expressivity theory: Genuinely interesting. RWKV-7's beyond-TC⁰ result and the delta rule's overwrite mechanics are the first credible theoretical cracks in attention's dominance, not just engineering wins.

Takeaway#

The transformer's monopoly is over in the practical sense that matters most: for long-context serving, the frontier has moved to hybrid architectures where attention is a minority component. Mamba proved selective SSMs can match transformers; RWKV-7 proved recurrences can out-theorize them; Gated DeltaNet became the linear-attention layer that shipping products are built on. Pure attention isn't going away — the hybrids keep it around for a reason — but the era of "transformer everywhere, full attention in every layer" is ending. The next architecture debate won't be transformer versus challenger. It'll be about the ratio.