Cloud and provider evals
Anthropic evaluation guide
Anthropic's evaluation guidance explains how to define measurable success, start with 20 to 50 realistic tasks, run repeated trials, grade outcomes and traces, combine code, model, and human graders, and turn stable capability checks into regression suites.
- Maintainer
- Anthropic
- Deployment
- Documentation, cookbooks, and sample code; no managed runner
- Status
- Current as of September 22, 2026
What it does
Anthropic's evaluation guidance explains how to define measurable success, start with 20 to 50 realistic tasks, run repeated trials, grade outcomes and traces, combine code, model, and human graders, and turn stable capability checks into regression suites.
- Tasks, trials, graders, traces, outcomes, evaluation harnesses, and agent harnesses
- Code-based, model-based, and human grading with outcome-oriented checks
- Capability and regression suites, pass@k, pass^k, partial credit, judge calibration, and CI-ready workflows
Where it fits
Teams designing task-specific Claude and agent evaluation suites with realistic cases, clear success criteria, calibrated graders, and regression gates.
Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.
Limitations and cautions
- This is guidance, not a runner, dashboard, dataset store, monitoring service, or SLA-backed product.
- Teams must operate the harness, persistence, CI, monitoring, and human review, and must calibrate qualitative judges.
- Anthropic retired the legacy Workbench evaluation workflow in August 2026, so old Console-eval tutorials no longer describe the current product.
How to evaluate the evaluator
Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.
Start with 20 to 50 realistic tasks
Use manual checks, support cases, production failures, bug reports, and costly near misses. Give each task a clear request, a known acceptable outcome, and an isolated starting state so one trial cannot contaminate the next.
- Keep capability tasks difficult enough to expose improvement.
- Move consistently solved behavior into a regression suite.
- Measure quality beside latency, tokens, tool errors, and cost.
Grade outcomes first, then inspect the trajectory
A rigid expected tool sequence can reject a valid solution. Grade the final state when several routes are acceptable. Inspect and grade the trace when tool choice, ordering, permissions, safety, cost, or the user's experience is part of success.
- Use deterministic code for exact, checkable facts.
- Use model judges for rubric-based qualities and calibrate them against expert labels.
- Keep human review for ambiguous, high-risk, or subjective cases.
Repeat trials for stochastic agents
pass@k asks whether at least one of k attempts succeeds, while pass^k asks whether every attempt succeeds. The second measure is stricter and more useful for workflows where a failed attempt is costly or a user cannot safely retry. Always report k and preserve failed traces.
- Freeze the model, harness, tools, data, permissions, and budgets for comparisons.
- Allow an Unknown judge result when evidence is insufficient.
- Replace saturated capability tasks while keeping them as regression checks.
Run the suite in development, CI, and production
Use the same task definitions and grader contracts across local experiments and release gates. Sample production traces to find failures the offline suite missed, review those failures, and promote representative cases into the next regression set.
- Block releases only with checks whose false-pass and false-fail rates you understand.
- Version prompts, models, tools, datasets, graders, and environment fixtures together.
- Track score changes by failure type rather than relying on one aggregate average.
Videos that explain the workflow
These are supplementary explanations. Use Anthropic's guide and cookbooks as the primary implementation references.
Common questions
How are agent evals different from prompt evals?
Prompt evals often judge one response. Agent evals must also account for state, tool calls, retrieval, intermediate decisions, errors, stopping behavior, and the final environment outcome.
How many evaluation cases do I need to start?
Start with 20 to 50 realistic tasks. Add cases as manual review and production use reveal new failure modes.
Should I grade the final answer or the trace?
Grade the outcome when multiple valid paths exist. Grade the trace when process quality, permissions, safety, cost, or tool choice matters.
When should I use code, an LLM judge, or a human?
Use code for deterministic facts, a calibrated model judge for scalable rubric decisions, and people for ambiguous or high-risk judgment. Strong suites usually combine all three.
What is pass@k versus pass^k?
pass@k asks whether any of k attempts succeeds. pass^k asks whether every attempt succeeds, making it a stricter reliability measure.
Can I still use the old Anthropic Workbench eval UI?
No. Anthropic sunset legacy Workbench in August 2026. Use a code-owned harness and current Anthropic cookbooks instead.