Benchmarks and learning resources
Terminal-Bench
Terminal-Bench is a continuous benchmark of agent-model pairs on containerized terminal tasks with programmatic outcome verification. Harbor runs repeated trials and publishes versioned leaderboards.
- Maintainer
- Harbor Framework and Laude Institute
- Deployment
- Open benchmark run through Harbor
- Status
- Current as of September 22, 2026
What it does
Terminal-Bench is a continuous benchmark of agent-model pairs on containerized terminal tasks with programmatic outcome verification. Harbor runs repeated trials and publishes versioned leaderboards.
- Terminal tasks spanning coding, systems, networking, security, dependencies, science, and interactive tools
- Containerized environments, programmatic verifiers, repeated trials, and parallel execution
- Versioned releases, public leaderboards, oracle validation, and task-quality review
Where it fits
Comparing agents on difficult, verifiable coding, systems, security, and scientific work inside sandboxed terminals.
Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.
Limitations and cautions
- Results are comparable only when the exact release, harness, agent, model, resources, and trial count are reported.
- Terminal-Bench 2.1 corrected 28 of 89 version-2.0 tasks, showing sensitivity to dependencies, resources, networking, and sandboxes.
How to evaluate the evaluator
Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.