Practical guide · Agent evaluation
How to build an AI agent test set without a benchmark
Start with real work and expected final states. Add cases whose labels you can prove by construction: known past decisions, planted failures, removed requirements, irrelevant changes, and templates with controlled values. Keep the source, transformation, expected outcome, grader, and risk on every row. Have an expert review ambiguous labels, then run variable agents several times.
Published October 23, 2026 · Reviewed October 23, 2026 · 13 minute read
The short answer
When no public benchmark matches your agent, build a small versioned suite around the decision you need to make. Use production-like inputs, isolated tools, and a final state that code can inspect. Create synthetic cases only when you control the change and know how the expected outcome should respond. Review labels before tuning, preserve a held-out set, and add real failures after release.
What each test case must record
An agent case needs more than a prompt and a reference answer. The agent may call tools, change external state, ask a question, stop safely, or take several valid routes. Record enough context to replay the task and explain the verdict.
| Field | What to store | Why it matters |
|---|---|---|
| Initial state | User, records, files, permissions, clock, and available tools | Makes the case replayable |
| Task | The exact request and relevant policy | Defines what the agent may and must do |
| Expected outcome | Final state, required facts, refusal, escalation, or question | Gives the grader a checkable target |
| Allowed variation | Alternative valid routes and wording | Avoids grading one preferred trajectory as the only answer |
| Provenance | Source case, transformation, author, review status, and version | Shows whether the label deserves trust |
| Risk | Impact, reversibility, affected users, and required approval | Sets repetition and release thresholds |
Anthropic’s agent evaluation guide distinguishes the full trace from the final environment outcome. Store both, but grade the trace only where tool choice, order, permission, cost, or interaction quality affects the user.
A six-step workflow
- Define the decision and final state. Write the release question first. Name the database fields, files, messages, approvals, or refusal that prove success.
- Collect seed cases from real work. Start with 20 to 50 accepted tasks, known failures, policy edge cases, and costly near misses. Remove secrets and personal data before the suite leaves its controlled environment.
- Create cases with known labels. Start from a valid outcome and work backward to an input. Use approved historical decisions, filled templates, or a sandbox state you created yourself.
- Add controlled transformations. Plant one violation, remove one required clause, change an irrelevant detail, paraphrase the request, or change a value that should alter the answer. Write the expected relation before running the agent.
- Validate labels and graders. Use code for state, schemas, totals, tool arguments, and policy rules. Ask experts to review ambiguous cases and compare any model grader with their held-out labels.
- Run repeatedly and version the suite. Repeat nondeterministic tasks, report both average success and consistency, freeze a held-out set, and publish a new dataset version when reviewed production failures arrive.
OpenAI’s evaluation best-practices guide warns against generic metrics, unrepresentative datasets, unsupported intuition, and automated grading that has not been calibrated against people.
Five useful case-generation patterns
1. Answer first
Start from an expected answer or state, then generate or select the request that should produce it. A contract template with controlled values can define the expected fields. A seeded support database can define the only eligible refund. Reject generated cases when the request does not clearly imply the label.
2. Plant one failure
Copy a valid case and add one forbidden action, policy violation, missing approval, stale source, unsafe tool argument, or duplicate side effect. One mutation keeps the expected verdict explainable and helps you test whether the grader finds the intended fault.
3. Remove one requirement
Delete a required clause, record, source, or permission. The expected behavior may change from act to ask, refuse, or escalate. This pattern tests whether the agent notices missing evidence instead of filling the gap with a plausible assumption.
4. Change something irrelevant
Paraphrase the request, reorder independent documents, rename an internal identifier, or add unrelated metadata. The final decision should stay the same. A change reveals unwanted sensitivity when the expected invariant does not hold.
5. Replay a real failure
Copy the original state into a sandbox, replace sensitive values, and write the corrected outcome with the domain owner. Preserve the old failing trace for diagnosis, but grade fresh runs against the corrected state rather than against the agent’s explanation of itself.
Worked example: a supplier-contract review agent
Suppose an agent checks supplier contracts against a policy and opens a review ticket. Begin with 12 contracts that lawyers have already decided: four approved, four rejected, and four escalated. Store the policy version, contract text, expected decision, exact reasons, permitted tools, and expected ticket state.
Create a copy of an approved contract and plant one prohibited auto-renewal term. The expected decision changes to reject, while every other field stays fixed. Remove the governing-law clause from another copy and expect escalation. Change the supplier’s display name without changing the legal entity and expect the decision to remain stable. Fill a reviewed template with controlled notice periods and calculate the expected policy result in code.
Run each case five times in an isolated account. Grade the saved ticket and contract state first. Check that no unapproved message was sent and that the agent did not call a forbidden tool. A lawyer reviews disagreements and a random sample of passes. Keep several reviewed contracts hidden until the team has finished changing prompts and tools.
Measure consistency, not one lucky run
An agent that succeeds on one attempt can still fail often enough to harm users. The τ-bench paper compares final database state with an annotated goal and introduces pass^k, the probability that every one of k trials succeeds. Use repeated trials for tasks with variable reasoning, user simulation, retrieval, or tool order.
Choose k from the real use case. A low-risk drafting helper may tolerate retries. A payment or account-change agent may need every trial to pass, along with idempotency and approval checks. Report the number of cases and trials beside the rate so readers can judge the evidence.
Limitations and common mistakes
- Synthetic ease: generated requests often look cleaner than production language and omit awkward state.
- Label leakage: the prompt, filename, or planted wording can reveal the intended answer.
- Stale truth: a known answer stops being correct when policy, data, tools, or time changes.
- Shared blind spots: one model can generate, solve, and grade cases with the same mistaken assumption.
- Trajectory overfitting: requiring one exact tool sequence can reject a different valid route.
- Easy averages: many simple cases can hide failure on rare, costly work. Report risk slices separately.
- Unsafe replay: production cases must run in an isolated environment with scrubbed data and blocked side effects.
NIST’s AI Risk Management Framework connects evaluation with risks to people, organizations, and society. Let impact and reversibility determine which cases block a release and which only trigger review.
Test-set checklist
- □ Every case names a checkable final state or required escalation.
- □ Each synthetic label follows from a written controlled transformation.
- □ Sensitive production data has been removed or kept inside its approved boundary.
- □ Code grades state and invariants wherever possible.
- □ Experts reviewed ambiguous labels and a sample of passes.
- □ Variable tasks run more than once.
- □ High-risk slices have their own release thresholds.
- □ The dataset, harness, tools, model, and grader versions are recorded.