Inspect AI

What it does

Inspect AI is an MIT-licensed Python framework for composing datasets, solvers, tools, agents, scorers, and metrics. It runs hosted or local models and can place code-executing work in Docker or external sandboxes.

• Composable tasks, solvers, tools, agents, scorers, and metrics
• Hosted API and local-model support with evaluation logs and parallel execution
• Docker and external sandboxes plus a catalog of more than 200 evaluations

Where it fits

Rigorous, reproducible model and agent capability or safety evaluations involving tools, multi-turn interaction, coding, or sandboxed environments.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• Its center of gravity is model and agent benchmarking rather than prompt management or production monitoring.
• The built-in local sandbox is not isolated, and large suites can require substantial model, compute, and sandbox resources.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.