AssemblyAI ranks 50+ LLMs on real voice tasks: the best picks are mostly the cheap ones
AssemblyAI released Voice Standard Bench today, a public benchmark that scores more than 50 language models on the four tasks voice products actually run through an LLM — summarization, information extraction, text correction, and dictation — with quality, speed, and cost reported side by side. The main finding, in AssemblyAI's own words: most voice tasks don’t need frontier models, or frontier-model prices.
Most voice products do more than produce a transcript. After speech-to-text returns the text, the next step is usually an LLM call: cleaning up dictation, summarizing a call, pulling out details, or drafting a follow-up. At volume, AssemblyAI says, that step quickly becomes a large share of the bill — often paid at frontier-model prices for tasks that don’t need a frontier model. The benchmark arrives alongside an expansion of AssemblyAI’s LLM Gateway to more than 50 models, all available behind the same API key customers already use for speech-to-text, including fast, low-cost open models the company hosts on its own infrastructure.
What it tests
Voice Standard Bench scores models on the inputs voice products actually send: real speech-to-text output, filler words, false starts, and the occasional misheard word included, at production-typical lengths. Every model gets the same transcripts and the same prompt, at four lengths — about 30 seconds, 2 minutes, 15 minutes, and 60 minutes.
Two judge models, Claude Opus 5.5 and GPT-5.6 Sol, score each answer against a fixed rubric per task, with scores tracked separately so the offset between the two judges can be measured, then combined. Every metric is normalized to a 0–100 scale; dictation is scored mechanically against known-correct output, with no judge at all. The bench also tracks the unglamorous stuff: format failures, refusals, timeouts, and error rates for every model-task combination. Models that fail the reliability check still appear in the charts with their scores, but they don’t get recommended.
The take-home picks
AssemblyAI publishes a grid of picks per task and transcript length — top quality, lowest price, and lowest latency, with the price and latency picks required to land within 8 points of the top score so a cheap model that fails the task can’t win the column. The headline results from the current run:
Summarization: Claude Opus 5.5 leads at 98 quality, but GPT-6 Luna hits 94 at the lowest price, and GLM-5.3 Flash delivers 91 at the lowest latency. Information extraction: Claude Opus 5 tops at 87, GPT-6 Luna again takes the budget slot at 86, GPT-4.1 is fastest at 79. Text correction: three OpenAI models — GPT-5.6 Sol, GPT-6 Astra, and GPT-6.1 Sol — tie at 91, while the open-weight gemma-4-31b reaches 84 at the lowest price. Dictation: Claude Opus 4.6 and Gemini 3.5 Flash tie at 99, gemma-4-31b lands 97 on a budget. Weighted across all tasks, GPT-6 Astra scores 92 overall, GPT-6 Luna 90 at the lowest price, and GPT-4.1 84 at the lowest latency.
The number that should make CTOs reach for a calculator: in the current run, the top scorer costs 55 to 65 times more per hour of audio than a model within two quality points of it. Only on hour-long transcripts does that premium narrow, to about five times.
Where small models win — and where they don’t
Open models win on speed for short transcripts. On a 30-second voicemail, gemma-4-31b answers in under a second at 93 quality, against GPT-6 Luna’s 1.7 seconds; on a two-minute call it answers in 1.5 seconds at 89 versus 2.4. Qwen3.5 4B Fast is the fastest model in the run at every length up to 15 minutes. That said, as transcripts approach meeting length, larger models pull ahead on quality and the speed gap closes.
Surprisingly, open models rarely win on cost. GPT-6 Luna, at its default reasoning setting, scores within a point of the best open model at every transcript length while costing between 0.4 and 1.6 cents per hour of audio — the open models that undercut it save fractions of a cent and, past two minutes, give up at least 10 quality points to do it.
Run it on your own transcripts
Voice Standard Bench is open source: developers can download the code from GitHub, point it at their own transcripts, and run a private version with the same tasks, rubrics, and judges, scored on their own data. For those who don’t want to run code, there’s a Playground compare mode that runs one prompt against several models side by side, and “Try in Gateway” links on the results page that open the playground with that model and task prompt already loaded.
The honest caveat comes from AssemblyAI itself: its transcripts aren’t yours. Different accents, domain vocabulary, prompts, and cost limits can change which model works best, so the bench is a starting point for which two or three models to test — not the final answer. Results refresh as new models launch on the Gateway, with each run dated.
Sources: AssemblyAI — “Introducing Voice Standard Bench: 50+ models, ranked by performance, speed and cost on real audio” (October 13, 2026); AssemblyAI — Voice Standard Bench results page; Blockchain.News — “AssemblyAI Launches Voice Standard Bench to Rank 50+ LLMs on Real Audio Tasks” (October 13, 2026).