MILO-BENCH

LONG-HORIZON SOFTWARE EVOLUTION

Can frontier models perform long-horizon software evolution with production-grade reliability?

by Ethara AI

01 · Overview

MILO-Bench trains AI agents to perform long-horizon software evolution extended from SWE-EVO. Given real-world codebases and their development history, the system works through sequences of 2 to 100+ consecutive production pull requests across Python, Rust, Go, TypeScript, JavaScript, Java, C, and C++. Unlike single-issue setups, it requires maintaining coherent, consistent development over extended timelines. Each instance reconstructs a milestone from post-training-cutoff repositories, with hermetic Docker environments.

  • Long-horizon evaluation: 2–100+ consecutive PR sequences per instance
  • Multi-language coverage across 8 programming languages
  • Contamination-audited instances from post-training-cutoff repositories
  • Hermetic Docker environments with private holistic test oracles
  • Three-model reference evaluation framework for reproducible scoring

02 · Key metrics

Three numbers that define the scope of MILO-Bench.

Instances Curated

10,000

across 8 languages

Models Evaluated

3

Claude Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro

Trajectories Generated

90,000

3 models × 3 seeds × 10,000 instances

03 · Methodology

How the metrics and scores are computed. Four principles govern the MILO-Bench scoring system.

Principle 01

Long-Horizon Software Evolution

  • Tasks span multi-step, multi-commit evolution chains across entire repositories
  • Each trajectory requires understanding of full codebase context and history
  • 90,000 trajectories generated across 10,000 instances

Principle 02

Production-Grade Reliability

  • Patches must pass CI test suites - no regressions allowed
  • Cross-file consistency verified across the full dependency graph
  • Build integrity maintained through the entire evolution chain

Principle 03

Multi-Language Coverage

  • 8 programming languages: Python, JavaScript, TypeScript, Java, C++, Go, Rust, Ruby
  • Real-world repositories with production codebases
  • Difficulty calibrated by trajectory length, files touched, and dependency complexity

Principle 04

Three-Model Evaluation

  • Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro evaluated head-to-head
  • Each model gets three attempts per instance (Pass@3)
  • Three seeds per model - 90,000 trajectories for reproducible scoring

04 · Evaluation pipeline

The method

Three phases turn raw repositories into scored long-horizon software evolution framework.

Phase 01

Environment Setup

  • Crawl 500+ repos, identify post-cutoff PR sequences
  • Build JSONL dataset with instance metadata
  • Construct hermetic Docker images per instance (base → PR image)
  • Generate run results and test baselines

Phase 02

Agent Interaction

  • Run AI agents in sandboxed Docker environments
  • 3 models × 3 seeds × 10,000 instances
  • Agents receive issue context and produce git diff patches
  • Tool-use interaction across long-horizon sequences

Phase 03

Evaluation & Scoring

  • Apply model patches to clean repo checkouts
  • Evaluate against full test suite in hermetic containers
  • Produce pass@k summaries per instance
  • Grade resolution verdicts across all trajectories

05 · Results

Pass@3 by difficulty tier for three frontier models (Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro).

All models clear Trivial tasks, but pass rates fall off sharply through Medium and Hard — reaching 0% at the Expert tier.

Fig. 1 - Pass@3 by difficulty tier (Trivial to Expert)

Distribution of best Pass@3 rate (highest of Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro).

18 ≥ 50% PASS 36.0%
20 10–49% PASS 40.0%
8 < 10% PASS 16.0%
4 0% PASS 8.0%
39.3% Mean Pass Rate

06 · Dataset Viewer

Browse the 10,000 tasks

Filter and search the full delivery. Expand a row to see per-model, per-run results and links to the raw trajectory and instruction file.

Total Instances

10,000

Models

3

Total Runs

270

Mean Pass Rate

39.3 %

Loading…
Task Codebase Language Tier Src hunks Pass rate Mean score

07 · Model comparison

Head-to-head breakdown of all three evaluated frontier models on long-horizon evolution tasks.

Claude Opus 4.8

Overall Pass Rate
38.6%
Short Trajectories
51.2%
Long Trajectories
24.8%
Avg Cost/Instance
~$0.07

GPT-5.5

Overall Pass Rate
36.4%
Short Trajectories
49.7%
Long Trajectories
23.1%
Avg Cost/Instance
~$0.06

Gemini 3.1 Pro

Overall Pass Rate
31.2%
Short Trajectories
44.5%
Long Trajectories
18.9%
Avg Cost/Instance
~$0.04

08 · Resources

GitHub

Harness

github.com/Ethara-Ai/aurora

HuggingFace

Dataset

huggingface.co/datasets/ethara/MILO-Bench

Paper

Research Paper

arxiv.org/abs/2512.18470