Dataset
Representative tasks, edge cases, costly failures, references, metadata, and production examples.
Learnetto field guide · Updated September 22, 2026
AI evals: a practical guide to evaluating LLM apps and agents
Evals turn “this version feels better” into evidence. This guide explains datasets, graders, regression suites, RAG and agent evaluation, then compares 26 current frameworks, platforms, cloud services, benchmarks, and learning resources.
What are AI evals?
An eval is a repeatable experiment: give a defined version of an AI system representative inputs, capture its outputs and relevant traces, apply one or more graders, and compare the result with a baseline or release threshold. The system under test includes more than a model. Prompts, retrieval, tools, context management, sampling settings, infrastructure, and agent harnesses can all change the outcome.
Useful evals are product-specific. Public benchmarks can reveal broad model capabilities, but they do not answer whether your support agent resolves the right ticket, your research agent cites primary sources, or your coding agent changes the correct files without regressions.
Harness
The exact prompts, tools, retrieval, environment, budgets, versions, and execution path being tested.
Graders
Deterministic checks, model judges, human review, or combinations that map outputs and traces to scores.
Build an eval suite in seven steps
- 1. Define the decision. State what the eval will decide: release readiness, prompt selection, model routing, regression detection, or incident diagnosis.
- 2. Write observable success criteria. Replace “good answer” with facts, actions, constraints, user outcomes, and unacceptable failure modes.
- 3. Start with real cases. Collect 20 to 50 accepted tasks, failures, difficult edge cases, and high-consequence scenarios. Tag each by task and failure type.
- 4. Choose the least subjective grader that works. Use code for schemas, tool arguments, state, citations, and executable outcomes; reserve model judges and humans for qualities code cannot capture.
- 5. Validate graders. Compare automated labels with expert labels, inspect false passes and false failures, and measure disagreement by category.
- 6. Run matched experiments. Hold the harness, tools, budgets, timeouts, and data constant. Repeat nondeterministic tasks and report confidence or trial distributions.
- 7. Close the loop. Turn production failures into regression cases, keep a held-out set, version every input, and review raw traces when aggregate scores change.
Choose the right grader
Code-based graders are fastest and most reproducible. Use tests, exact or fuzzy matching, schema validation, state inspection, citation verification, policy engines, and task-specific calculations whenever success is mechanically checkable.
Model-based graders can score relevance, style, grounded synthesis, conversation quality, and other open-ended criteria. Give them narrow rubrics, examples, structured outputs, and an abstain path. Validate them against domain experts before trusting the score.
Human review remains necessary for ambiguous, high-stakes, subjective, or changing requirements. Use annotation guidance, blinded comparisons, multiple raters where needed, and adjudication for disagreements.
Offline evals versus online evals
Offline evals replay a controlled dataset before release. They are suited to version comparisons, capability hill-climbing, regression gates, and reproducible debugging.
Online evals score or sample production traces after release. They detect drift, unknown failures, safety incidents, and changes in real user traffic—but often lack reference answers.
Use both. Offline suites prevent known regressions; production monitoring discovers what the suite missed. The handoff between them is the core improvement loop.
How to evaluate agents
- Grade final state and task completion before prescribing one “correct” trajectory.
- Measure tool selection, argument correctness, permissions, recovery, stopping behavior, latency, tokens, and external cost where they matter.
- Run repeated trials and report pass@k for discovering at least one success or pass^k when every attempt must succeed.
- Control sandbox resources, network access, timeouts, harness versions, and provider settings so infrastructure noise does not masquerade as capability.
How to evaluate RAG
- Score retrieval separately from generation: context precision, recall, ranking, freshness, and source authority.
- Check claim-level support, citation correctness, answer completeness, refusal when evidence is missing, and robustness to conflicting documents.
- Use realistic corpora, access controls, difficult negatives, unanswerable questions, and changed documents—not a clean toy index.
- Inspect failures by component so a generator is not blamed for missing retrieval, or a retriever for a bad judge.
Common eval mistakes
One vanity score
A global average hides catastrophic failures and improvements limited to one task type. Report slices and raw examples.
Unvalidated LLM judges
A confident numeric score is not evidence until judge decisions agree with representative expert labels.
Benchmark overfitting
Keep a held-out set and refresh tasks. Public questions, leaked answers, and repeated prompt tuning can invalidate results.
Changing multiple variables
If model, prompt, tools, timeout, and retrieval all change together, the eval cannot explain the result.
Ignoring variance
Repeat stochastic and agentic tasks. A one-run leaderboard can reward luck or infrastructure conditions.
Testing only happy paths
Include rare costly failures, adversarial inputs, missing context, tool errors, and cases that should be refused or escalated.
Scoring outputs but not outcomes
For agents, inspect the final environment and user result—not only polished text or an apparently sensible trace.
Never reading traces
Aggregate metrics identify movement; transcripts and artifacts explain why the system changed and whether the grader is wrong.
OpenAI Evals shutdown notice
OpenAI says its hosted Evals product will no longer be available after November 30, 2026. It remains documented today, so it is included for existing users and migration research. Do not choose it as a new long-term dependency; prefer an actively supported, code-owned, or portable evaluation workflow.
Curated directory
26 eval tools and resources
Selection favors official documentation, active workflows, distinct use cases, and practical fit. A listing is not an endorsement; each page names deployment trade-offs and limitations.
Open-source frameworks
OpenAI / Promptfoo
Promptfoo
Promptfoo is a local-first CLI and Node.js library for testing LLM applications from configuration files. It combines quality evaluation, provider comparison, CI gates, and adversarial red teaming without requiring teams to move their test suite into a hosted platform.
MIT-licensed local CLI and library; optional sharing and enterprise services
Read the full profile →Confident AI
DeepEval
DeepEval is an Apache-2.0 framework for end-to-end, component, conversational, RAG, multimodal, and agent evaluation. It offers a pytest-like workflow and an optional Confident AI service for shared dashboards and production monitoring.
Apache-2.0 local framework; optional Confident AI cloud
Read the full profile →Vibrant Labs
Ragas
Ragas is an Apache-2.0 Python library centered on RAG evaluation. It separates retrieval and generation quality with metrics such as context precision, context recall, faithfulness, and answer relevance, while also supporting agent and traditional text metrics.
Apache-2.0 local Python library
Read the full profile →UK AI Security Institute
Inspect AI
Inspect AI is an MIT-licensed Python framework for composing datasets, solvers, tools, agents, scorers, and metrics. It runs hosted or local models and can place code-executing work in Docker or external sandboxes.
MIT-licensed local framework with optional sandbox backends
Read the full profile →LangChain
OpenEvals
OpenEvals is an MIT-licensed evaluator library with model-graded, deterministic, RAG, safety, tool-call, code, image, voice, and trajectory evaluators. It runs locally and can optionally log experiments to LangSmith.
MIT-licensed Python and TypeScript packages
Read the full profile →Giskard AI
Giskard
Giskard v3 is an Apache-2.0, async-first framework for scenario tests, natural-language judges, groundedness checks, and vulnerability scans. It targets agents and chat systems rather than the tabular-model focus of the older v2 product.
Apache-2.0 local Python packages
Read the full profile →Evaluation platforms
Arize AI
Arize Phoenix
Phoenix combines LLM and agent tracing with versioned datasets, experiments, LLM-based evaluation, retrieval evaluation, and prompt tooling. It can be self-hosted through Docker or Kubernetes and also connects to Arize's managed products.
ELv2 source-available self-hosting; managed Arize products available
Read the full profile →Braintrust Data
Braintrust
Braintrust combines offline experiments, production observability, prompt comparison, datasets, scorers, and online evaluation. It supports managed cloud and a hybrid self-hosted data plane for teams with stricter data requirements.
Managed cloud, BYOC, or hybrid self-hosted data plane
Read the full profile →LangChain
LangSmith
LangSmith supports offline and online evaluation with code checks, human review, pairwise comparisons, and LLM judges. Production traces can be sampled, scored, inspected, and added back to datasets as regression cases.
Managed SaaS; Enterprise self-hosting add-on
Read the full profile →ClickHouse
Langfuse
Langfuse is a self-hostable AI engineering platform with traces, prompt management, dataset experiments, LLM judges, code evaluators, annotation queues, and production scores. ClickHouse acquired Langfuse in January 2026 while retaining its MIT-licensed core.
Managed Langfuse Cloud or MIT-licensed self-hosting
Read the full profile →Weights & Biases
W&B Weave
Weave adds tracing and repeatable evaluations to the W&B stack. Evaluation objects connect datasets, model or application functions, scorers, per-example results, multiple trials, latency, and cost.
W&B cloud, dedicated cloud, or Kubernetes-based self-managed deployment
Read the full profile →Comet
Opik
Opik connects traces, datasets, evaluation experiments, natural-language assertions, prebuilt metrics, custom scoring, optimization, and annotation. Production failures can be promoted into repeatable test cases.
Comet-managed cloud or open-source self-hosting
Read the full profile →MLflow Project
MLflow GenAI evaluation
MLflow's GenAI stack stores datasets, expectations, outputs, traces, scorers, human feedback, and experiment results in the familiar MLflow tracking model. Databricks offers additional managed capabilities, while core MLflow remains open source.
Open-source self-hosting; managed offerings vary by provider
Read the full profile →Patronus AI
Patronus AI
Patronus offers evaluator families such as GLIDER, Judge, and Lynx alongside experiments, datasets, tracing, monitoring, simulation, and red teaming. It targets organizations that need specialized reliability checks rather than only generic prompt comparisons.
Managed platform or Kubernetes-based self-hosting
Read the full profile →Cloud and provider evals
OpenAI
OpenAI Evals
OpenAI Evals defines tasks with test-data schemas and graders, runs asynchronous dataset evaluations, and reports row-level results. It is available as of September 22, 2026 but is being wound down and is not a durable choice for a new evaluation stack.
Hosted OpenAI platform; available but scheduled for shutdown
Read the full profile →Google Cloud
Vertex AI Gen AI evaluation service
Vertex AI's evaluation service supports pointwise and pairwise model-based metrics, deterministic metrics, function-call checks, custom computation, and agent trajectory evaluation. It accepts data from pandas, Cloud Storage, or BigQuery.
Managed Google Cloud service
Read the full profile →Microsoft
Microsoft Foundry evaluations
Microsoft Foundry evaluations run in the portal or SDK over datasets, traces, and simulated conversations. The evaluator catalog spans quality, similarity, RAG, risk and safety, agents, rubrics, and Azure OpenAI graders.
Managed Microsoft Foundry service and SDK
Read the full profile →Amazon Web Services
Amazon Bedrock Evaluations
Bedrock evaluation jobs can compare models, prompts, knowledge bases, and external RAG sources using built-in metrics, custom datasets, judge models, or human workers. Jobs run through the console, CLI, or SDK and write reports to S3.
Managed Amazon Bedrock service
Read the full profile →Anthropic
Anthropic evaluation guide
Anthropic's guide explains how to define measurable success criteria, assemble task-specific and held-out datasets, include edge cases, combine metrics, and automate grading. It is a methodology resource rather than a hosted evaluation product.
Documentation and sample code; no managed runner
Read the full profile →Benchmarks and learning resources
Harbor Framework Team
Harbor
Harbor packages instructions, environments, agents, tests, logs, and rewards into reproducible trials. It integrates with popular coding agents, runs locally in Docker or on cloud sandboxes, and is the official harness for Terminal-Bench.
Open-source local and cloud-scale harness
Read the full profile →EleutherAI
Language Model Evaluation Harness
EleutherAI's harness provides a unified task format and execution layer for dozens of standard benchmarks and hundreds of variants. It supports local and API models, custom tasks, adapters, batching, caching, and distributed inference.
Open-source Python benchmark runner
Read the full profile →Hugging Face
Lighteval
Lighteval is Hugging Face's multi-backend evaluation toolkit with thousands of tasks and sample-level output inspection. It can execute through Inspect AI, Accelerate, Nanotron, vLLM, SGLang, TGI, LiteLLM, and Hugging Face services.
Open-source local and distributed toolkit
Read the full profile →SWE-bench team
SWE-bench
SWE-bench turns real open-source issues into reproducible repository tasks and grades candidate patches in Docker. The family includes Original, Lite, Verified, Multilingual, and Multimodal variants plus official leaderboards.
Open-source benchmark and Docker evaluation harness
Read the full profile →Harbor Framework and Laude Institute
Terminal-Bench
Terminal-Bench is a continuous benchmark of agent-model pairs on containerized terminal tasks with programmatic outcome verification. Harbor runs repeated trials and publishes versioned leaderboards.
Open benchmark run through Harbor
Read the full profile →Anthropic
Demystifying evals for AI agents
Anthropic's field guide explains tasks, trials, graders, transcripts, outcomes, capability suites, regression suites, repeated trials, and evaluation maintenance. It recommends combining code, model, and human graders while reading raw traces.
Official methodology article
Read the full profile →Hamel Husain and Shreya Shankar
LLM Evals: Everything You Need to Know
This living FAQ presents an opinionated workflow for trace review, error analysis, failure taxonomies, evaluator datasets, code checks, validated LLM judges, RAG evals, and production feedback.
Living article and course material
Read the full profile →Frequently asked questions
What are AI evals?
AI evals are repeatable tests that run an AI system on representative inputs, apply explicit grading rules to its outputs or behavior, and produce evidence that can be compared across prompts, models, data, tools, and releases.
How many examples do I need to start?
Start with 20 to 50 high-value cases drawn from real tasks and known failures. Add cases as production traces reveal new failure modes; do not wait for a perfect thousand-row benchmark before measuring anything.
Should an LLM grade another LLM?
Sometimes. LLM judges are useful for open-ended qualities, but they must be calibrated against human labels. Prefer deterministic checks whenever the requirement can be verified with code, state, tests, or exact facts.
What is the difference between an eval and monitoring?
Offline evals compare controlled versions before release. Online evals and monitoring sample real production traces after release. Mature teams connect both so a production failure becomes a permanent regression case.
What should an agent eval measure?
Measure final task success first, then inspect tool choice, arguments, trajectory, safety, latency, cost, and recovery behavior where those details affect users or risk.