LLM Evals: Everything You Need to Know

What it does

This living FAQ presents an opinionated workflow for trace review, error analysis, failure taxonomies, evaluator datasets, code checks, validated LLM judges, RAG evals, and production feedback.

• Minimum viable eval setups, trace review, error analysis, and failure taxonomies
• Domain-expert labels, pass/fail criteria, code evaluators, and validated LLM judges
• Evaluator datasets, dev/test splits, RAG evaluation, synthetic data, monitoring, and A/B tests

Where it fits

Practitioners building product-specific evals from traces, domain-expert judgments, and observed failures instead of relying on generic benchmarks.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• The authors explicitly present sharp opinions rather than universal rules.
• The process is human-intensive: reliable failure discovery and judge validation require domain expertise and hand-validated labels.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.