Finding 01
Precise Log-Sigmoid Scaling
- Averaged performance follows
S(t) = S_max / (1 + (t_mid / t)^β) - Holds across the full task suite and all six capability families
- Late performance is forecastable from early trajectories
01 · Overview
Pretraining scaling laws revealed that model capability improves predictably with data and compute. But once agents are deployed, they must learn from interaction with real-world environments, and whether that learning obeys any clean scaling law was, until recently, unknown.
Mephisto distills the empirical and theoretical findings behind a large-scale study of agent and environment interaction across many real-world tasks. Averaged agent performance during environment learning follows a precise log-sigmoid scaling law, S(t) = S_max / (1 + (t_mid / t)^β), and frontier agents' learning speed is roughly doubling every three months.
- Precise log-sigmoid scaling across the full suite and every capability family
- A theoretical derivation: frontier expansion on latent task graphs
- Learning speed doubling roughly every three months across model generations
- Dual-loop feedback: a local inner loop and a judge-mediated outer loop
- Perception-first tasks (GUI operation) excluded so learning is not conflated with perception
02 · Key metrics
Three numbers that define the scope of the Mephisto benchmark and the learning dynamics it measures.
Capability Families
6
diverse task domains
Task Horizon
12h+
continuous operation
Learning Speed
2×
every ~3 months
03 · Methodology
What the study found. Four findings govern how agents learn from their environments.
Finding 02
A Theoretical Derivation
- Environment learning modeled as frontier expansion on latent task graphs
- Explains why benchmark-averaged progress takes the log-sigmoid form
- Connects observed curves to an underlying process, not a curve fit alone
Finding 03
Learning Speed Doubling ~Every 3 Months
- Across recent frontier model generations, learning speed scales rapidly
- The rate at which agents learn from environments roughly doubles quarterly
- A fast-moving axis distinct from raw pretraining capability
Finding 04
Learning Dynamics Dominate Long Horizons
- How agents use accumulated experience matters more than attempt count
- Continuous experience beats independent restarts; longer context aids retention
- Feedback converts many failed probes into a few durable gains
04 · Evaluation pipeline
The method
A dual-loop protocol turns rich, multi-level feedback into an authoritative, submission-gated grade.
Phase 01
Inner Loop (local)
- A writable workspace with compilers, linters, simulators, and docs
- Self-split validation data for fast local checks
- Unlimited fast iteration, entirely agent-driven
Phase 02
Outer Loop (judge-mediated)
- Submitted artifacts graded by a hidden judge container
- Private test cases, unseen seeds, or client-style rubrics
- Slower, authoritative, and submission-gated
Phase 03
Host-Side Judge Server
- Mediates queues, cooldowns, and authentication
- Runs asynchronous grading against private evaluation harnesses
- Agents keep working while long evaluations run
05 · The scaling law
Averaged agent performance during environment learning is fit precisely by a log-sigmoid curve in elapsed interaction time.
The same functional form, S(t) = S_max / (1 + (t_mid / t)^β), holds across the whole task suite and within every individual capability family, where S_max is the asymptotic ceiling, t_mid the transition midpoint, and β the sharpness of the ramp.
Every figure below is a task-bundle fact read straight from the released Mephisto corpus metadata; no model rollout or reward is used.
4
SYSTEMS & SOFTWARE
80.0%
1
PROFESSIONAL KNOWLEDGE
20.0%
5
SAMPLE CORPUS
released tasks
80%
of the sample corpus is Systems & Software Engineering (4 / 5)
06 · Dataset Viewer
Total Tasks
5
Families
2
Base Images
2
Judge Timeout
600–900s
| Task | Family | Base Image | Submit Files | Eval Timeout | Repo |
|---|
07 · Family span
The two capability families represented in the released sample corpus, side by side. Each dials a different point on the task-domain gradient.
Systems & Software Engineering · 4 tasks
- Base images
- python · rust
- Mean eval timeout
- 600s
- Submit files
- 1–2
- Horizon
- ≥ 12h
- Score direction
- maximize
Professional Knowledge Work · 1 task
- Base image
- python
- Eval timeout
- 900s
- Submit paths
- strategy/
- Horizon
- ≥ 12h
- Score direction
- maximize
08 · Resources
GitHub
mephisto
github.com/Ethara-ai/mephisto
Hugging Face
mephisto
huggingface.co/datasets/ethara/mephisto/tree/main
Need a custom agent environment or eval harness?
Ethara builds held-out, reproducible agent environments and datasets for frontier teams.