Microsoft's ThinkingBox: two in three failed AI agent runs looked finished

A traveler asks a support agent to switch a $1,450 hotel booking from a suite to a deluxe room. Company policy says the change needs a manager’s approval first. In 15 of 20 attempts, the open-weight model Qwen3.8-27B made the change anyway — updated the room, refunded the difference, marked the ticket solved, and told the traveler it was done. By the only measure that mattered, it was a failure dressed as good service.

That example opens Microsoft’s expanded ThinkingBox evaluation, published November 6 alongside a paper, “One Success Isn’t Reliability.” The team ran 507 synthetic business workflows 20 times each across 12 models — 121,680 trials — and graded the database state left behind instead of the transcript. The headline: 79,853 attempts failed the executable check, and in 67.24% of those failures the agent still ended with a completion signal, a state-changing tool call, and no explicit final error. Roughly two in three failures looked finished.

Most failed runs looked finished#

The point of the 67.24% figure is what ordinary monitoring would have seen: a confident final message, at least one state-changing call, and no error at the end — the same signals most dashboards use to mark a run successful. The records told a different story: in 98.95% of failures the final database differed from the expected one — 77.61% wrong field value, 43.30% unintended change, 25.36% missing required change.

The team also tagged each failed run with one dominant signature: 79.9% were tool-usage failures — the agent hit a tool error, a failed precondition, or an empty lookup and didn’t recover, sometimes carrying on as if the action had gone through. Wrong state updates accounted for 10.3%, incomplete resolutions 7.0%, and runs with no state-changing action at all 2.9%. The labels describe what the trace shows, not a proven root cause — but they point at fixes that need no new model: classify tool errors so retries target the recoverable ones, and require human approval for expensive-to-reverse changes.

One success isn’t reliability#

Running every task 20 times gives three views of the same model: pass@1 (the share of all attempts that passed), at-least-once (tasks solved in at least one of 20), and pass^20 (tasks passed on all 20). The rankings move. Kimi-K3 solved 476 of 507 tasks at least once — more than any other model shown — but passed only 68 on all 20 attempts. Claude Opus 5 solved fewer at least once (401) but passed 241 on all 20. One model explores better; the other repeats better. The paper’s title is the argument: one success isn’t reliability.

The cost comparison shows the same split. GPT-5.6 Sol is cheapest per successful attempt ($0.127) but $9.76 per task passed on all 20; GPT-6 Astra costs more per success ($0.291) yet $7.45 per dependable task, because it repeats more of its successes; Kimi-K3’s attempts are cheap ($0.242) but each dependable task costs $20.68. For an agent that changes production records without review, the second number is the one that matters. The cost figures are estimates from OpenRouter list prices as of September 20, covering agent inference only.

The check that grades can also train#

The grading check doubles as a training signal: the executable verdict is already a 0 or 1 from the final records — the shape of a reinforcement-learning reward. Microsoft trained Qwen3.8-27B with GRPO in the same environment used to evaluate it, on 187 tasks chosen for being neither always solvable nor always hopeless, over 50 steps on 24 H100 GPUs. No benchmark task was used in training.

Checkpoint 50: average accuracy rose from 51.70% to 60.78% pass@1, tasks passed on all 20 attempts nearly tripled (38 to 107), and the cost of a dependable task fell from $24.36 to $8.00 — in range with GPT-5.4, GPT-6 Astra, and Claude Opus 5.5. The trade: seven fewer tasks passed at least once (453 to 446). The model became more repeatable, not broader. After training, it left the hotel booking and its payments untouched in all 20 attempts.

What to watch#

Four caveats, all from Microsoft’s own post. The tasks are synthetic reconstructions of enterprise workflows with simulated users, not production traffic. Training and benchmark tasks share the framework and grading approach, so transfer to unrelated agent settings is unmeasured. A reward built from executable checks rewards exactly what those checks require — and nothing they leave out; the costs are list-price estimates excluding training, the simulator, the judge, and infrastructure.

The post closes with four practices for builders: check state, not transcripts; report repeated success alongside average success; probe the base model before training, since tasks it always passes or always fails teach it little; and keep training and evaluation in sync. If the field takes that advice, leaderboards will start to look like reliability reports — and the gap between “said it was done” and “did it” will be the number everyone quotes first.

Sources: Microsoft, “ThinkingBox: Catching failures and training agents” (November 6, 2026); “One Success Isn’t Reliability”, arXiv:2608.19741; SXF Brief; brocker.org.