Supermicro KV cache storage with IBM and NVIDIA: 56x faster first tokens on 130,000-token prompts

Supermicro announced the solution on November 5, 2026, positioning it for the workloads that now dominate enterprise AI: agentic systems, coding assistants, multi-turn chat and long-context retrieval-augmented generation. Those applications build up enormous working memory as they chew through codebases, documents, images and conversation history — often more than GPU and CPU memory can hold. The new system gives that memory somewhere persistent to live, shared across GPU nodes and clusters instead of being recomputed or discarded.

How it works

Running a model has two phases. During prefill, the input text is converted into tokens and processed to produce intermediate key-value (KV) states — the model’s working memory of the context. During decode, the model generates its response using those states. The catch is that the KV states for a long prompt are expensive to compute, and if a follow-up request reuses the same context, the system normally computes them all over again.

The joint solution stores previously computed KV states in a persistent cache on shared flash storage, so later requests can reuse them instead of recomputing. NVIDIA’s Dynamo inference framework adds cache-aware routing, separates prefill from decode, and moves cached blocks through its KV Block Manager and the NIXL transfer library. IBM Storage Scale Erasure Coding Edition supplies the shared storage tier — the G4 tier in NVIDIA’s memory hierarchy — while Supermicro’s Petascale servers provide the all-flash hardware, running on AMD EPYC 9005 Series (“Turin”) processors, and Spectrum-X Ethernet moves data between the pieces. If you want the underlying mechanics in more depth, our explainer on KV caches and PagedAttention covers the memory engineering behind fast serving.

What the testing showed

The headline figure comes from joint Supermicro and NVIDIA testing with a 120-billion-parameter model: persistent KV cache delivered up to 56 times faster time-to-first-token at 130,000-token prompts. As input length grew from 30,000 to 130,000 tokens, time-to-first-token increased by only 0.377 seconds with the cache, compared with 29.55 seconds without it — the point being consistency as much as speed, since long-context workloads otherwise get slower exactly when they get bigger.

The deeper numbers come from the IBM Redbooks reference architecture, Context without Limits: A High-Performance KV Cache Platform for Large-Scale AI Inference, which the launch is built on. At 130,000 tokens it reports 32.14 seconds for GPU recomputation versus 0.570 seconds using the storage-backed cache. A separate 200-request run — 28 concurrent connections, 100 unique prompts, 24 million tokens and 825 GB of KV cache data — saw throughput rise from 0.19 to 4.26 requests per second, with total processing time falling from 1,048.56 to 46.94 seconds. The cache started empty, so that average includes a cold-start period while reuse built up.

The testing also covered the messy conditions real deployments face. Under a noisy-neighbor test, four other clients generated 200 GB/s of competing network traffic; the cached system still sustained 3.6 requests per second and finished 200 requests in 55.56 seconds, an 18% throughput reduction from its clean-network result. All inference measurements came from one Supermicro GPU client node with four NVIDIA RTX PRO 6000 GPUs, connected to an eight-node Petascale storage cluster, running Dynamo v0.9.0+, vLLM v0.14.1, NVIDIA AIPerf and the openai/gpt-oss-120b model, with GPU-memory and system-DRAM cache tiers disabled to isolate the shared storage tier. Three Spectrum-X SN5600 switches formed the test network; client-facing bandwidth reached up to 80 GB/s, while the eight-node storage system achieved up to 315 GB/s of read throughput.

What’s inside the box

The solution is four products wired together rather than a single new device: Supermicro Petascale all-flash storage servers; IBM Storage Scale software; NVIDIA Dynamo; and NVIDIA Spectrum-X Ethernet switches. The companies say the integrated architecture was validated across high-concurrency and noisy-neighbor scenarios, and that Supermicro’s factory rack integration and on-site deployment services can cut deployment time from weeks to days. Sizing guidance in the Redbook uses four storage nodes for small proof-of-concept deployments, eight for the tested medium configuration, and 16 or more for large AI factories.

“As AI workloads expand from training to inference, enterprises and neoclouds deploying high-throughput inference workloads characteristic of agentic AI and long-context, multi-turn generative AI need to remove bottlenecks to improve GPU efficiency,” said Charles Liang, president and CEO of Supermicro. Sam Werner, general manager of IBM Storage, said the collaboration was “an important step toward making enterprise AI inference more scalable, reliable, and cost-effective,” while NVIDIA’s Shruti Koparkar, AI Inference Product Lead, said Dynamo “helps orchestrate distributed inference so enterprises can get more usable throughput from every GPU.”

The limits

The 56x figure belongs to a specific configuration — one model, one prompt mix, one concurrency level, one network — and the companies disclosed no pricing, orders or revenue tied to the product. The collaboration itself is not new: the three companies have worked on related efforts since mid-2025, the reference architecture was published in June 2026, and the platform falls under NVIDIA’s Inference Context Memory Storage Platform initiative, introduced at CES 2026. What is new today is the packaged, jointly validated product.

That framing matters for buyers. The launch is best read as a deployment question, not a benchmark: whether an organization’s actual context-reuse demand justifies a shared storage tier, and whether its client bandwidth can deliver what the storage can read — the Redbook notes client-side networking, not storage read capacity, was the configuration’s constraint. For teams running long-context inference at scale, though, the direction of travel is clear. Scality’s on-prem AI Inference Factory made a similar bet this autumn, and the labs’ race to secure gigawatts for their data centers is pushing the whole industry toward squeezing more inference out of every GPU already installed.

Sources: the Supermicro press release on PR Newswire, Unite.AI’s report, and the IBM Redbook reference architecture.