Milo-Bench measures long-horizon software-engineering capability, not just isolated coding ability. It evaluates whether an agent can complete milestone-scale engineering tasks that span multiple files, languages, and architectural components while keeping an existing production codebase correct — not the individual functions or short bug fixes that narrower benchmarks target. Where SWE-bench and Multi-SWE-bench score single-issue fixes, Milo-Bench targets whole milestones resolved across a codebase over a long agent trajectory.
- Milestone-scale tasks spanning multiple files and architectural components
- Multi-language coverage across 8 programming languages, 19 codebases
- Continuous partial-credit scoring so long-task progress stays visible
- Network-sealed verifier — reference solutions cannot influence a run
- Three frontier models (Claude Opus 4.8, Gemini 3.1 Pro, GPT-5.6 Sol) × 3 runs each
src_hunks).
Five outcome-based tiers
Tiers are computed from observed model performance across runs. Trivial tasks are solved by nearly every run; Expert tasks by none. This is what the models actually experienced, not any property fixed in advance.
Head-to-head, three frontier models
Pass rate captures how hard the tasks are, not how far apart the models are. Mean score keeps crediting partial progress where pass rate bottoms out — this can even flip the model ranking. Claude Opus 4.8 leads on both metrics, but Gemini 3.1 Pro and GPT-5.6 Sol trade places: GPT-5.6 Sol is last by pass@3 (36.7% vs 37.8%) yet clearly ahead by mean score (55.8% vs 47.8%), because it often gets most of the way without landing an exact 1.0.
| Model | Provider | Runs | Pass@3 | Mean score | Total cost |
|---|
Where the models differ
| Tier |
|---|
| Tier |
|---|
Network-sealed, continuous, partial credit
The verifier computes a continuous score per run:
score = recall × regression_factor. A run solves a task only when
score is exactly 1.0. The verifier runs network-sealed, so the reference solution
cannot influence a run.
Task
instruction.md + Docker environment
Agent
long tool-use trajectory
Patch
edits across files & languages
Verifier
network-sealed target tests
Score
per-run graded result
8 languages across 19 codebases
Milo-Bench spans eight programming languages drawn from nineteen production codebases; each language card summarises how many of the 30 tasks land in that language.
Browse the 30 tasks
Filter and search the full delivery. Expand a row to see per-model, per-run results and links to the raw trajectory and instruction file.
| Task | Codebase | Language | Tier | Src hunks | Pass rate | Mean score |
|---|