OpenEvals

What it does

OpenEvals is an MIT-licensed evaluator library with model-graded, deterministic, RAG, safety, tool-call, code, image, voice, and trajectory evaluators. It runs locally and can optionally log experiments to LangSmith.

• Customizable LLM-as-judge evaluators and reusable rubric prompts
• Exact match, edit distance, embedding, structured-output, tool-call, and code checks
• Agent-trajectory scoring, sandboxed-code evaluation, and multi-turn user simulation

Where it fits

Python or TypeScript developers who want composable evaluator functions without adopting a full evaluation platform.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• It is a starting toolkit rather than a dataset store, dashboard, or complete evaluation platform.
• Many evaluators need a judge model and credentials; managed experiments and feedback storage depend on LangSmith.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.