Lighteval

What it does

Lighteval is Hugging Face's multi-backend evaluation toolkit with thousands of tasks and sample-level output inspection. It can execute through Inspect AI, Accelerate, Nanotron, vLLM, SGLang, TGI, LiteLLM, and Hugging Face services.

• Thousands of tasks across knowledge, reasoning, math, code, long context, chat, and languages
• Custom tasks, models, metrics, scoring, and detailed sample-by-sample outputs
• Inspect AI, Accelerate, distributed inference, API backends, and local or remote result storage

Where it fits

Teams evaluating local, distributed, or API-served models across a large multilingual task catalog while retaining sample-level results.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• Windows is officially untested and unsupported, and not every task works with the preferred Inspect AI backend.
• Backend-specific paths and evolving task implementations require strict version pinning for meaningful comparisons.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.