Demystifying evals for AI agents

What it does

Anthropic's field guide explains how to turn real agent behavior into tasks, repeated trials, outcome checks, trace review, calibrated graders, capability suites, and regression gates. Start with 20 to 50 realistic cases rather than waiting for a perfect benchmark.

• Tasks, trials, graders, transcripts, outcomes, eval harnesses, and agent harnesses
• Code-based, model-based, and human grading with outcome-oriented checks
• Capability versus regression suites, pass@k, pass^k, partial credit, and grader calibration

Where it fits

Product and engineering teams designing practical evaluations for multi-turn, tool-using agents.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• This is methodology, not executable software or a benchmark suite.
• Automated evals create false confidence when tasks do not match real use, and LLM judges still require calibration against human decisions.
• Anthropic retired the legacy Workbench evaluation workflow in August 2026, so use a code-owned harness and current cookbooks rather than old Console tutorials.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.

Build the first suite from real failures

Collect 20 to 50 tasks from manual checks, support cases, production traces, bug reports, and expensive near misses. Give each task an unambiguous request, a known acceptable outcome, and an isolated starting state so another run does not inherit files, messages, or database changes from the previous one.

• Keep capability tasks difficult enough to reveal improvement.
• Move consistently solved behavior into a regression suite that should remain near 100%.
• Record quality, latency, token use, tool errors, and cost for every trial.

Choose the cheapest grader that can judge the claim

Use code for exact fields, state changes, permissions, links, and other deterministic facts. Use model graders for rubric-based qualities that cannot be reduced to one exact answer, then test those graders against expert labels. Keep human review for ambiguous, high-risk, or genuinely subjective cases.

• Allow an Unknown result when the evidence is insufficient.
• Read false passes and false failures instead of trusting one agreement percentage.
• Grade the final outcome by default; inspect the trajectory when tool choice, order, safety, cost, or user experience matters.

Measure stochastic agents with repeated trials

One successful run does not establish reliability. pass@k asks whether at least one of k attempts succeeds, which is useful when a user may retry. pass^k asks whether every one of k attempts succeeds, which is closer to the reliability needed for unattended or high-cost workflows. Report the trial count beside either metric.

• Freeze the model, harness, tools, permissions, data, and budgets for comparisons.
• Store the complete trace and final environment state for failed runs.
• Treat a saturated capability suite as a regression suite and replace it with harder capability tasks.