Cloud and provider evals
OpenAI Evals
OpenAI Evals defines tasks with test-data schemas and graders, runs asynchronous dataset evaluations, and reports row-level results. It is available as of September 22, 2026 but is being wound down and is not a durable choice for a new evaluation stack.
- Maintainer
- OpenAI
- Deployment
- Hosted OpenAI platform; available but scheduled for shutdown
- Status
- Available but scheduled for shutdown on November 30, 2026
What it does
OpenAI Evals defines tasks with test-data schemas and graders, runs asynchronous dataset evaluations, and reports row-level results. It is available as of September 22, 2026 but is being wound down and is not a durable choice for a new evaluation stack.
- Dataset schemas, asynchronous runs, automated graders, and human annotations
- Trace grading, prompt optimization, third-party model support, and row-level analysis
- Pass/fail counts, per-criterion results, usage reporting, API, and dashboard workflows
Where it fits
Existing OpenAI API teams maintaining dataset-based prompt or model regression tests during the remaining service window.
Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.
Limitations and cautions
- OpenAI says Evals will no longer be available after November 30, 2026.
- Do not adopt it as a new long-term dependency; plan a migration to a code-owned or actively supported evaluation runner.
How to evaluate the evaluator
Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.