What this video shows
The speakers start from the gap between an agent saying it finished and the work actually being finished. They turn goals into checks the agent can run: command-line health checks, build and test commands, and browser snapshots that confirm a page renders. A failing check returns to the agent, which then makes another attempt.
They add LLM judges and evaluation baselines for qualities a script cannot measure, using a live demo application. The design principle is to make everything the agent can verify return a machine-readable result. The talk is a vendor workshop for an authentication product, so its platform claims reflect that product. The loop itself is portable: cheap deterministic checks first, judges only where a script cannot decide.
Use Playwright snapshot testing for the browser checks the talk relies on. Read Anthropic's test and evaluation guidance for turning a goal into a scored check.
What you will learn
- Give the agent goals it can check itself, so a failure returns to the agent before it reaches a person.
- Prefer deterministic checks such as tests and health endpoints, and reserve judges for qualities they cannot measure.
- Capture browser snapshots so a visual regression fails a build rather than a review.
- Keep a baseline so a change that lowers scores is visible before merge.
How to apply this safely
- Turn one agent goal into a command whose exit code says pass or fail.
- Add a browser snapshot for the user-facing part of the change.
- Return the check output to the agent and let it retry until the check passes or it reports a blocker.
- Add a baseline and run the checks on every change so regressions are caught automatically.
Important limitations
- The workshop promotes WorkOS products, so its tooling choices are vendor guidance.
- A passing check proves only what the check measures, so a weak test can still pass a wrong implementation.
- Judge-based checks add their own variance and cost, so they belong after cheaper deterministic checks.
Sources to check
- Playwright snapshots Snapshot testing for browser-based checks.
- Anthropic evaluation guidance Turning goals into scored checks and test cases.
Continue learning on Learnetto
Review AI-generated code
Use a six-pass review before agent output reaches main.
Sandbox AI coding agents
Bound filesystem, network, and credentials for the agent's checks.
AI evals guide
Build a baseline and release gate around the checks.