Harbor

What it does

Harbor packages instructions, environments, agents, tests, logs, and rewards into reproducible trials. It integrates with popular coding agents, runs locally in Docker or on cloud sandboxes, and is the official harness for Terminal-Bench.

• Containerized agent tasks with instructions, environments, test scripts, logs, and rewards
• Integrations for Claude Code, Codex CLI, OpenHands, Gemini CLI, and other agents
• Custom and composite datasets, cloud scaling, RL rollouts, prompt optimization, and SFT workflows

Where it fits

Running coding and computer-use agents against reproducible, sandboxed task suites at local or cloud scale.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• Agentic runs can be slow and expensive; substantial experiments often require horizontal cloud scaling.
• Scores depend on pinned dataset, agent, model, environment, and harness versions, and container providers can differ.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.