What this video shows
Weill describes replacing a slow expert-only review loop with answer-first test generation. For a contract-review agent, he proposes cases from past decisions, planted policy violations, removed clauses, irrelevant changes, and templates whose blanks define the expected values.
The method works when you control the transformation and can prove the label. Keep the original and transformed case together, record the rule that produced the expected outcome, and send ambiguous samples to an expert. Add real production failures because synthetic cases will miss language, state, and workflow conditions you did not model.
Learnetto's agent test-set guide expands the answer-first pattern into a versioned dataset workflow. OpenAI's evaluation best practices recommend representative production cases and human calibration.
What you will learn
- Start from an expected answer or state when the label must be trustworthy.
- Use one controlled mutation per synthetic case so the reason for failure stays clear.
- Prefer code and state checks when success can be verified mechanically.
- Reserve expert time for ambiguous labels, new failures, and high-risk decisions.
How to apply this safely
- Choose one narrow task with a policy or state you can encode.
- Create seed cases from accepted decisions and known failures.
- Generate planted, removed, irrelevant, and template-filled variants with written labels.
- Run expert review on a sample and add production misses to the next dataset version.
Important limitations
- Generated inputs can be easier and cleaner than real user requests.
- A known answer can still be wrong when the underlying policy or source is stale.
- Synthetic transformations cover only the failure types their designer anticipated.
- Air Canada employs the speaker, and the talk reports a team method rather than a controlled comparison.
Sources to check
- Evaluation best practices OpenAI guidance on representative datasets, explicit criteria, and human calibration.
- How to Test AI Agents When No Benchmark Exists AI Engineer video published October 22, 2026.
Continue learning on Learnetto
Build an AI agent test set
Use answer-first, seeded, and invariant cases with reviewable labels.
AI evals guide
Choose deterministic, model, and human graders.
Verify AI citations
Check whether supplied evidence supports each claim.