Testing agents: task-based evals for tool-using models

Checking an agent's outputs for "good vibes" works until it starts calling tools across many turns — then mistakes propagate, creative workarounds slip past sloppy checks, and "it felt fine" stops meaning anything.

Task-based evals are the fix: give the agent a concrete task in a controlled environment, then apply explicit grading logic to what it did. This tutorial covers the four pieces that matter — sandboxes, graders, success criteria, and failure taxonomies — drawing on field-tested practice, including Anthropic's engineering write-up on evaluating agents.

A task is one test with defined inputs and success criteria; a trial is one attempt at it; a grader is logic that scores some aspect of performance; the transcript is the complete record of a trial; the outcome is the final environment state; the harness runs trials end to end; a suite is a collection of tasks.

The transcript-versus-outcome distinction is the one that bites beginners. A booking agent might announce "your flight is booked," but the outcome is whether a reservation actually exists in the database. Grade the transcript when you care about how the agent behaved; grade the outcome when you care about what got done. Most serious suites grade both.

1. Sandboxes: the environment is half the eval#

An eval is only as trustworthy as its environment. Each trial should start from a clean, isolated environment, and the eval agent should function roughly the same as the production agent.

Why isolation matters:

• Shared state creates correlated failures. Leftovers or resource exhaustion from one trial can break the next — and suddenly you're measuring infrastructure flakiness, not agent quality.
• Shared state can inflate scores. In one internal eval, a model gained an unfair advantage by reading git history left behind by previous trials. Fresh container, fresh repo, every time.
• Noisy environments make results uninterpretable. Trials failing from the same CPU or memory limit aren't independent measurements of the agent.

For coding agents, that means containers with pinned dependencies and a fresh repo per trial. For computer-use agents, a virtualized OS with scripts that inspect the resulting state. For conversational agents, a simulated user persona plus a backend database the agent must update. Containerized harnesses — Harbor, Inspect AI, Braintrust, LangSmith, the open-source Langfuse — are now the standard.

2. Graders: code, model, and human#

Agent evals typically combine three grader types, each with a clear job:

Choose deterministic graders where possible, LLM graders where necessary, and humans for calibration. Two hard-won lessons:

1. Grade what the agent produced, not the path it took. Requiring exact tool-call sequences in exact order is brittle and punishes agents that find valid approaches you didn't anticipate — like the model that "failed" a flight-booking task by discovering a policy loophole that was genuinely better for the user. Grading creativity out is a failure of the eval, not the agent.
2. Calibrate LLM judges against humans. Give the judge a way out — an instruction to return "Unknown" when evidence is insufficient — and grade each dimension with an isolated judge rather than one judge scoring everything.

Per-task scoring can be binary, weighted, or hybrid.

3. Success criteria: what "pass" actually means#

Capability evals vs. regression evals. Capability evals ask "what can this agent do well?" and should start at a low pass rate — a hill to climb. Regression evals ask "does it still handle everything it used to?" and should sit near 100%; any decline signals breakage. Graduate saturated capability evals into the regression suite. SWE-bench Verified is the cautionary tale: frontier-model scores climbed from roughly 40% to over 80% in about a year, at which point it measures reliability, not frontier capability.

pass@k vs. pass^k. pass@k is the probability of at least one success in k attempts — right when one good answer is enough. pass^k is the probability that all k trials succeed — right for customer-facing agents where users expect reliability every time. Pick the one that matches your product, and be explicit about k.

Write reference solutions and partial credit. Each task should be passable by an agent that follows instructions correctly, and you should prove it by writing a reference solution that passes every grader — this catches the classic failure where the task description and the grading disagree. For multi-component tasks, build in partial credit: a support agent that diagnoses the problem but fumbles the refund is meaningfully better than one that fails immediately.

The warning worth keeping on your desk: one frontier model scored 42% on a reproduction benchmark until researchers fixed rigid grading, ambiguous specs, and stochastic tasks — after which it scored 95%. Always debug the eval before blaming the agent.

4. Failure taxonomies: what actually went wrong#

For the tool-use dimension specifically, four failure modes capture nearly everything that pure-conversation evals miss — and they're scored independently, because improving one often trades off against another:

The same four dimensions sit inside a broader class-level taxonomy for whole trials:

• budget_exhausted — token/iteration budget hit before any grading. Fix: raise the cap or shorten the loop.
• true_task_fail — the grader ran and returned failure: genuine agent behavior (task logic, tool choice, API usage).
• grader_bug — the eval itself is broken: mismatched API names, redacted placeholders in tool args, thresholds that punish instruction-following.
• harness_blocker — infra failures: spawn errors, missing tool registrations, stalls in unattended runs.
• partial_success — real progress, wrong final state. Partial-credit graders should catch and classify these.

The diagnostic rule: when scores don't climb, read the transcripts before touching the agent. Failures should seem fair. If they don't, the eval is what needs the fix.

Putting it together: a minimal eval spec#

You don't need a platform to start. A spec file, a resettable container, and a runner script beat a sophisticated framework with mushy tasks. Anthropic's suggested starting point: 20–50 simple tasks drawn from real failures — enough because early agent changes have large effect sizes.

Here's the shape of one task:

task:
  id: "refund_angry_customer_1"
  desc: "A frustrated customer requests a refund for order #4821 ($74.99). Verify identity, process the refund, confirm by email."
  env: {sandbox: "docker://eval-sandbox:2026-09", seed_db: "fixtures/refunds.sql"}
  graders:
    - type: state_check          # outcome
      expect:
        refunds: {order_id: 4821, status: processed}
        tickets: {status: resolved}
    - type: tool_calls            # transcript: hard requirements only
      required: [{tool: verify_identity}, {tool: process_refund}]
    - type: transcript
      max_turns: 10
    - type: llm_rubric            # interaction quality
      assertions: ["Agent showed empathy for the customer's frustration", "Resolution was clearly explained"]
  metrics: [n_turns, n_toolcalls, n_total_tokens, cost_per_task]

Notice what it doesn't dictate: the tool-call sequence or the exact wording. It grades the outcome, a few hard tool-use requirements, a budget constraint, and interaction quality.

Takeaway: the eval-driven development checklist#

The checklist:

1. Start now. 20–50 tasks from manual checks, bug reports, and support tickets.
2. Isolate everything. Clean sandbox per trial; no shared state between runs.
3. Write unambiguous tasks with reference solutions. If two domain experts wouldn't reach the same pass/fail verdict, rewrite the task.
4. Combine grader types. Deterministic where possible, LLM judges where needed, humans for calibration.
5. Separate capability from regression suites. Low starting pass rate for the hill-climb; near-100% for the guardrails. Graduate saturated capability tasks into regression.
6. Classify every failure. A four-dimension tool-use score plus class-level labels turns red runs into a prioritized fix list.
7. Read the transcripts. Someone reads raw trial transcripts every week. Metrics are never trusted until a human has verified the grading is fair.
8. Guard against hacks. Design graders so passing requires solving the problem, not exploiting a loophole.

Teams without evals debug reactively: wait for complaints, reproduce manually, fix one thing, break another. Teams with them turn failures into test cases and test cases into regression suites — a shared definition of "good" the whole team can climb toward. The value compounds, but only if evals are infrastructure from the start, not a chore added later.