Together Compiler: AI agents that rewrite GPU code to make open models faster overnight

Together AI announced Together Compiler on November 4, 2026, a system that uses teams of AI agents to rewrite the GPU code behind the open-source models it serves. New models get faster overnight instead of over weeks, with no additional work from customers, according to the company. Together calls it the first step toward recursive self-improving inference — an inference stack that keeps tuning itself as models, workloads, and chips change.

The scale behind the claim matters: Together says the system already serves trillions of tokens a day. The problem it is aimed at is churn. Together cites OpenRouter traffic, saying 80% of it goes to models released in the last three months, and that each of the ten most popular open models on OpenRouter uses a different architecture.

Why inference tuning keeps expiring

Hand-tuning the low-level code that runs a model is specialist work — writing GPU kernels by hand takes expertise most teams do not have on staff. Agents can now do some of that work, but Together says validation is the bottleneck: in a traditional system, a single agent edit can take up to 15 minutes to validate in an inference engine. Together Compiler is designed to maximize how fast agents can iterate, running 252x more experiments than existing systems that use agents for inference optimization, while keeping costly GPUs fully utilized.

Most inference today uses only a fraction of what the GPU can actually do, Dan Fu, Together AI’s founding senior vice president of research, said in the announcement — and the gap opens back up every time a new model or chip comes out. Closing it for good requires inference that adapts just as quickly.

How it works: agents rewrite the code under the model

In Together’s description, agent teams continually rewrite and test the low-level GPU code — kernels — that runs a model, and the winning versions go straight into production. The loop runs again for each new model, workload, or chip. That low-level layer is where much of the speed lives, and where rivals are also working, as DeepSeek’s open-source DeepInfer inference engine shows.

Together Compiler is the first project of a larger initiative Together calls Polymorphic Inference, described as recursive self-improvement for inference workflows: inference that adapts on its own, with AI agents continuously improving the systems that run AI. Together says the approach reaches state-of-the-art optimizations in 24 hours and deploys them immediately to production.

The numbers Together is citing

On Kimi K3 and GLM 5.3, Together Compiler reached 1.5x and 1.4x higher peak throughput than the best result on SemiAnalysis’s InferenceX, an open inference benchmark, according to the announcement. On a 1B Llama model, it reached over 2,100 tokens per second and up to 1.8x the speed of hand-written megakernels, including Together AI’s own, and was fastest at every output length tested on both NVIDIA H100 and B200 GPUs. MiniMax M3.1 Flash reached 1,066 tokens per second on NVIDIA B200 GPUs, 1.5x the baseline, while DeepSeek v4.1 Flash reached 1,163 tokens per second on NVIDIA GB300, 1.2x the baseline.

Those are vendor benchmarks: Together’s numbers, from Together’s tests, against the baselines it names. They measure speed, and the announcement offers no independent evaluation. The commercial context is clear enough — inference demand is outrunning capacity, as Fal’s $1.3 billion raise at a $16 billion valuation shows — so throughput gains matter — and need verifying.

The one customer example in the announcement is Phonic, a voice AI company. “Running Together Compiler cut our median time to first speech by nearly 50%, and it happened overnight,” said Moin Nadeem, Phonic’s co-founder. Getting its Phonic-1 model optimized that quickly let the company ship sooner, he said; Phonic has been a Together customer for over two years.

What you can use today

Together Compiler is running today across many of Together AI’s most popular serverless endpoints, the company says. Customers can join a limited beta to use it on their own dedicated deployments, with self-serve access coming soon. Together has published an overview of Polymorphic Inference and a research paper on Together Compiler.

Together Compiler is not a new model and not a weights release. It optimizes how existing open models are served. The test now is whether the 24-hour loop holds for the next wave of models.