Milo-Bench measures long-horizon software-engineering capability, not just isolated coding ability. It evaluates whether an agent can complete milestone-scale engineering tasks that span multiple files, languages, and architectural components while keeping an existing production codebase correct — not the individual functions or short bug fixes that narrower benchmarks target. Where SWE-bench and Multi-SWE-bench score single-issue fixes, Milo-Bench targets whole milestones resolved across a codebase over a long agent trajectory.
- Milestone-scale tasks spanning multiple files and architectural components
- Multi-language coverage across 8 programming languages, 19 codebases
- Continuous partial-credit scoring so long-task progress stays visible
- Network-sealed verifier — reference solutions cannot influence a run
- Three frontier models (Claude Opus 4.8, Gemini 3.1 Pro, GPT-5.5) × 3 runs each
src_hunks).
Five outcome-based tiers
Tiers are computed from observed model performance across runs. Trivial tasks are solved by nearly every run; Expert tasks by none. This is what the models actually experienced, not any property fixed in advance.
Head-to-head, three frontier models
Pass rate captures how hard the tasks are, not how far apart the models are. Mean score keeps crediting partial progress where pass rate bottoms out — this can even flip the model ranking (GPT-5.5 leads by pass@3, Claude Opus 4.8 by mean score).
| Model | Provider | Runs | Pass@3 | Mean score | Total cost |
|---|
Where the models differ
| Tier |
|---|
| Tier |
|---|
Network-sealed, continuous, partial credit
The verifier computes a continuous score per run:
score = recall × regression_factor. A run solves a task only when
score is exactly 1.0. The verifier runs network-sealed, so the reference solution
cannot influence a run.
Task
instruction.md + Docker environment
Agent
long tool-use trajectory
Patch
edits across files & languages
Verifier
network-sealed target tests
Score
per-run graded result
8 languages across 19 codebases
Milo-Bench spans eight programming languages drawn from nineteen production codebases; each language card summarises how many of the 30 tasks land in that language.
Browse the 30 tasks
Filter and search the full delivery. Expand a row to see per-model, per-run results and links to the raw trajectory and instruction file.
| Task | Codebase | Language | Tier | Src hunks | Pass rate | Mean score |
|---|