What this video shows
Pane reports that a cheaper candidate disagreed with production on 18% of examples, while repeated production runs disagreed with each other on roughly 7%. His synthetic HarborBite example then shows a sequence: trace prompt versions and cost, preserve reusable prefixes for caching, use DSPy and GEPA to search for shorter instructions, and test a smaller model only after the task and scorer are explicit.
The illustrated optimizer removes 51% of instruction tokens, but Pane does not present it as proof of lower cost at unchanged quality. He recommends shadow evaluation across production versus itself, candidate versus itself, and candidate versus production. Reviewers label disagreements, calculate a paired confidence interval around net improvement, and inspect critical categories separately. Learnetto would also keep an untouched test set because an optimizer can fit the examples used during search.
Follow OpenAI's current prompt-caching guide to preserve stable prefixes and measure cached tokens, latency, and realized cost. Inspect the official DSPy GEPA documentation before defining the optimizer, metric, and evaluation budget.
What you will learn
- Measure prompt version, model, tokens, latency, cost, and outcome before changing the system.
- Estimate each model's own variation because raw model-to-model disagreement does not identify which answer is better.
- Use a separate test set after prompt search so examples that guided optimization do not also decide release.
- Review critical task categories separately because a strong overall average can hide damage to an important workflow.
How to apply this safely
- Choose the highest-cost bounded task and create train, development, and untouched test sets from representative requests.
- Run the current system several times per example to estimate variance and document ambiguous labels.
- Measure prefix caching first, then optimize instructions and test a smaller model one change at a time.
- Shadow both versions on paired requests, review disagreements, calculate uncertainty, and monitor accepted outcomes after release.
Important limitations
- The 18%, 7%, and 51% figures are speaker-reported examples from OutRival and a synthetic proxy. Learnetto did not inspect the datasets or reproduce the runs.
- A shorter prompt can omit rare safeguards, while a smaller model can overfit a narrow benchmark. Aggregate agreement alone will not expose every regression.
- Prompt caching behavior and prices vary by provider and model. Teams should use current provider documentation and their own usage records.
Sources to check
- OpenAI prompt caching Current cache eligibility, prefix, measurement, retention, and model-specific guidance.
- DSPy GEPA Official optimizer overview and configuration reference for the method named in the talk.
Continue learning on Learnetto
AI evals guide
Build datasets, compare versions, and define release gates.
Best AI resources for evals
Choose tools and methods for tracing, scoring, and review.
AI engineering courses
Study model behavior, systems, evaluation, and production trade-offs together.