Google Cloud's open AI-native Kubernetes stack: OpenRL, llm-d 1.0 and Agent Substrate for stateful agents

Every platform team running AI agents in production has hit the same wall: the demo works, then the bill and the breakage arrive. Multi-turn agents scatter their turns across random pods and rebuild their context from scratch each time. GPUs sit idle while a training run alternates between training steps and rollout generation. And one model-written shell command in a shared cluster is a sandbox escape waiting to happen. Google's blog names all four failure modes up front — accelerator underutilization, cache-blind load balancing, framework fragmentation, and agent security — and assigns one piece of the stack to each.

The five pieces, and what each one does#

The pieces interlock in one pipeline. OpenRL post-trains an open-weight model — Google uses its Gemma family as the example — with reinforcement learning, in plain Python, using four primitives: forward_backward, optim_step, sample and save_state. It time-slices training and rollout generation onto the same accelerators instead of dedicating separate clusters to each, and ships only parameter deltas to rollout workers instead of multi-gigabyte weight files after every optimization step. Then llm-d serves the result, KubeRay handles the distributed data and training jobs around it, and Agent Substrate plus gVisor run the agents themselves in isolated sandboxes.

The numbers Google is leading with#

llm-d is where the stack's numbers are loudest. Its Endpoint Picker — wired into the Kubernetes Gateway API Inference Extension and managed natively on GKE through GKE Inference Gateway — routes each request to the pod whose KV cache already holds that conversation. In production deployments, Google says it doubled KV cache hit rates from 35% to 70% and cut median time-to-first-token by 70–85% on multi-turn agent workloads. When accelerator memory fills up, a three-tier offload cascades KV blocks from high-bandwidth memory to host RAM to NVMe or cloud storage instead of recomputing context.

The proof point Google chose is a legal benchmark, not a chatbot demo. Google tested Gemma 4 against the Legal Agent Benchmark from Harvey, spanning 24 legal practice areas from M&A due diligence to contract disputes. Supervised fine-tuning plateaued around 50% accuracy on multi-step reasoning rubrics; reproducible RL loops with OpenRL lifted it to 70%. The whole pipeline shared GPU and TPU capacity with concurrent LoRA and inference jobs on GKE, with no manual partitioning.

Agent Substrate's numbers are about density and speed: idle agent sessions snapshotted to storage and restored in under 500 milliseconds, at 10x container density, according to Google. The project has drawn more than 100 contributors since public development started in May 2026. And llm-d itself is a coalition effort — co-founded by Google Cloud, Red Hat, IBM Research, CoreWeave and NVIDIA, with contributions from more than 500 engineers across 25-plus organizations.

Why this lands now#

The timing is KubeCon, and the context is a week in which the infrastructure race is visibly accelerating. Google announced two new eighth-generation TPU systems, TPU 8tx and TPU 8i Max, on November 9, and Red Hat took OpenShift 5 live the same day with its own agentic platform. The message from both vendors is the same: agents are moving from prototypes to multi-tenant production fleets, and whoever owns the plumbing owns the margin. Google's pitch is that the plumbing should be open — the stack runs on any conformant Kubernetes cluster, on CPUs, GPUs or TPUs, on any cloud — while the polished path runs on GKE.

That openness is also a standards play. Google donated Agent Substrate, gVisor and OpenRL to the Cloud Native Computing Foundation, where all three were accepted into the CNCF Sandbox last month. TorchTPU, meanwhile, went upstream into vLLM and SGLang — Google open-sourced it last month, co-developed with Meta's PyTorch team, so PyTorch models run natively on TPUs without private forks. The through-line: Google would rather set the open standard than sell you the closed runtime.

The catch#

Start with the numbers. Every performance figure in the announcement — the 35%-to-70% cache hit rate, the 70–85% time-to-first-token cut, the 50%-to-70% benchmark lift, the sub-500 ms restores — is Google's own, measured on Google's workloads. No independent benchmark of the stack exists yet, and the cautious read is to treat them as the shape of the win, not its size. The benchmark that produced the 70% figure is a legal test, not a general reasoning test; other domains may not lift as far.

Then the maturity. OpenRL is the newest layer "by a wide margin," in Google's own words — it started in February as a research preview, and Google says it donated the project early on purpose, so neutral governance is in place before the design hardens. That is honest, and it is also a warning: the design has not hardened. CNCF Sandbox is the foundation's entry level, and general availability applies to the managed Agent Substrate on GKE, not to the open-source projects themselves.

Finally, the stack is Google-shaped even where it is open. The Endpoint Picker is managed natively through GKE Inference Gateway; the reference hardware is Google's TPUs, with day-zero parity across TPU v5e, v6e and v7x. It will run elsewhere — that is the point of the donation — but the smoothest road runs through GKE, and teams adopting the open pieces self-managed should expect early-project rough edges.