Benchmarks and learning resources

SWE-bench

SWE-bench turns real open-source issues into reproducible repository tasks and grades candidate patches in Docker. The family includes Original, Lite, Verified, Multilingual, and Multimodal variants plus official leaderboards.

Open official resource Compare all eval tools
Maintainer
SWE-bench team
Deployment
Open-source benchmark and Docker evaluation harness
Status
Current as of September 22, 2026

What it does

SWE-bench turns real open-source issues into reproducible repository tasks and grades candidate patches in Docker. The family includes Original, Lite, Verified, Multilingual, and Multimodal variants plus official leaderboards.

  • Real GitHub issue-resolution tasks with executable repository tests
  • Docker-based grading, local and cloud evaluation, cached logs, and reports
  • Original, Lite, Verified, Multilingual, and Multimodal datasets and leaderboards

Where it fits

Measuring whether coding agents can resolve real GitHub issues by producing repository patches that pass executable tests.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

  • Local runs are resource intensive; official guidance recommends x86_64, 120 GB free storage, 16 GB RAM, and eight CPU cores.
  • Scores cover the benchmark's repository and issue distribution, not general software-engineering competence; ARM64 is experimental.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.

Primary sources