What this video shows
Jarmak separates model capability from the harness that supplies tools, context, retries, and stopping rules. She reports a prior benchmark where changing only the harness moved results by 30%, which makes a leaderboard a starting point rather than a production decision.
For your own system, write realistic tasks, repeat variable trials, preserve the full trace, and grade the outcome that matters. A quality gate must stop the release when it fails. Use production traces to discover missing cases, but keep sensitive data and unsupported agent self-reports out of the label.
Anthropic's agent eval guide separates traces from final outcomes and recommends repeated trials. Use Learnetto's agent test-set workflow to turn those principles into reviewable cases.
What you will learn
- Treat the model and harness as separate experimental variables.
- Use public benchmarks to shortlist candidates, then test the real workflow.
- Repeat trials and retain full traces when the agent can take different routes.
- Make a release gate block work instead of producing a dashboard nobody acts on.
How to apply this safely
- Name the decision and the user outcome the test must protect.
- Freeze the model, harness, tools, data, and budgets for each comparison.
- Run repeated trials and save tool calls, intermediate results, and final state.
- Review failures by task slice before allowing a release.
Important limitations
- The reported 30% shift comes from the speaker's benchmark work and does not predict the size of a harness effect in another system.
- Production traffic reveals realistic failures, but a live result can mix model, tool, data, and infrastructure causes.
- A trace can explain a failure without proving what the agent intended.
- Omni Analytics employs the speaker, and the talk is not an independent audit of Omni's agent.
Sources to check
- Demystifying evals for AI agents Anthropic's guide to tasks, trials, traces, outcomes, graders, and repeated runs.
- Don't Trust Your AI Agents AI Engineer video published October 23, 2026.
Continue learning on Learnetto
Build an AI agent test set
Create cases with known labels when no benchmark fits.
AI evals guide
Choose graders and regression gates.
Backtest AI agents
Replay historical decisions without future-data leakage.