Google open-sources TorchTPU, making TPUs a native PyTorch device — co-developed with Meta's PyTorch team

• Who: Google Cloud (Mark Lohmeyer, VP & GM of AI & Computing Infrastructure; Bill Jia, VP Engineering), announced October 20, 2026 at PyTorch Conference 2026.
• What: TorchTPU — an open-source native PyTorch backend for Google TPUs, co-developed with the PyTorch team at Meta; TPUs become a native PyTorch device (torch.device("tpu")).
• License and packaging: Apache 2.0 on GitHub; four repositories (TorchTPU, vLLM-TorchTPU, SGLang-TorchTPU, TorchTitan); v0.2.0 on PyPI, Python 3.11+.
• Training scale: up to 52% Model FLOPs Utilization on Qwen 3 8B with near-linear scaling to 8,192 Ironwood TPUs (TPU v7x); TorchTitan validated to 8,000 Ironwood.
• Serving: native PyTorch backends for vLLM (built with Inferact) and SGLang (built with RadixArk); llm-d 1.0 with Red Hat, IBM, 500+ contributors and 25 organizations (CNCF Sandbox).
• Ecosystem: Hugging Face Transformers v5 deploys to TPUs via device_map="tpu"; Zenteiq reported 4× throughput (1.47B → 5.97B tokens/day per 1,000 TPU v6e) on llm-d.

Google made its biggest play yet for the PyTorch developer on October 20, open-sourcing TorchTPU at PyTorch Conference 2026. The software — co-developed with the PyTorch team at Meta — makes Google's TPUs a native PyTorch device: existing training and serving code points at TPUs with minimal changes, debugs in eager mode, and compiles to production performance with torch.compile. In Google's framing, TPUs now sit alongside GPUs and CPUs as a first-class PyTorch target.

The move is really about the software moat around Nvidia's GPUs. CUDA's grip was never only silicon; it was that every training script, serving engine, and tutorial assumed one vendor, so moving meant rewriting. By making the framework and the serving stack people already run treat another accelerator as a native option, Google is attacking that switching cost directly — and it is not alone: a day earlier, Amazon announced its own native PyTorch backend for Trainium, according to The Diary of AI's October 20 briefing.

What TorchTPU is

TorchTPU is an open-source PyTorch backend from Google and Meta that runs PyTorch models natively on TPUs, plugging in through PyTorch's PrivateUse1 interface. Tensors are ordinary PyTorch tensors — no subclasses, no wrappers — with one process per device and preserved tensor views and in-place mutations. Debug in eager mode; hand the same model to torch.compile for production throughput, with Pallas kernels and distributed primitives built in. MarkTechPost, citing Google's numbers, reports the stack covers around 450 ATen operations and ships as v0.2.0 on PyPI for Python 3.11 and newer under the Apache 2.0 license.

The predecessor, PyTorch/XLA, recorded lazy-tensor graphs, and developers often had to add code marking compilable regions; it also supported only pure SPMD code, while TorchTPU handles divergent MPMD execution. MarkTechPost reports Google's team states TorchTPU replaces PyTorch/XLA and recommends it for all new workloads.

Serving: vLLM and SGLang go native

Serving engines get the same treatment. The native PyTorch vLLM backend on TorchTPU, developed with Inferact, deploys open-weight models on TPUs with continuous batching, quantization, and multi-chip parallelism — Inferact CEO Simon Mo said developers can run production workloads on TPUs with the throughput and latency they expect from the vLLM stack. The native SGLang backend, built with RadixArk, targets structured and agentic workloads using RadixAttention to reuse KV cache across requests; RadixArk CTO Banghua Zhu said TorchTPU gave a "painless path to native TPU acceleration."

For fleet-scale inference, Google released llm-d 1.0 alongside Red Hat, IBM, and a community of more than 500 contributors across 25 organizations; as a CNCF Sandbox project it routes requests prefix-cache-aware across GPU and TPU fleets. The numbers Google cites are striking: Zenteiq, using GKE's llm-d-powered Inference Gateway, reported more than 4× higher aggregate inference throughput — from 1.47 billion to 5.97 billion generated tokens per day per 1,000 TPU v6e accelerators.

Training at 8,192 Ironwood chips

On the training side, TorchTPU is integrated into TorchTitan, the PyTorch-native reference implementation for distributed training. TorchTitan carries recipes for Llama 3 70B/405B, Qwen 3 8B/32B, MuseGlimmer 30B, the Qwen 3 235B mixture-of-experts, and FLUX.1 12B — validated running on up to 8,000 Ironwood TPUs (TPU v7x). The headline benchmark: up to 52% Model FLOPs Utilization training Qwen 3 8B, with near-linear scaling across up to 8,192 Ironwood chips.

Post-training is covered too: Google upstreamed a TPU backend and a GRPO reference recipe to VeRL, the popular reinforcement-learning framework, and added native TorchTPU support to Ray so pipelines can mix CPUs, GPUs, and TPUs in one workflow. Anyscale chief architect Edward Oakes called the stack industry-standard — PyTorch, models, Ray, Kubernetes — with TPUs now embedded directly in it.

The ecosystem play

Google also partnered with Hugging Face: in Transformers v5, deploying models to TPUs is "as simple as passing device_map="tpu"," and Diffusers runs natively on TPUs with tensor parallelism. Meta is the loudest endorser, and its motive is transparent: PyTorch engineering director Jana van Greunen said Meta "cannot be constrained by a single hardware architecture," and that TorchTPU lets it push LLM, ads, and recommendation workloads onto Google silicon while feeding bugs back to both teams. Hugging Face VP of product Jeff Boudier framed it as hardware diversity for open-source AI. LiteRT (formerly TensorFlow Lite) joined the PyTorch ecosystem too, and a TPU agent suite for onboarding, diagnostics, and inference tuning is now in preview on GitHub.