Mephisto Samples publishes five reinforcement learning environments from the Professional Knowledge Work family. Each is a containerized workspace with a twelve-hour interaction window, a continuous reward in [0, 1], and a separate judge container holding evaluation assets the agent never sees.
Each task measures whether an agent improves across a long episode under repeated graded feedback instead of whether it produces one correct endpoint answer. The primary signal is a continuous scalar composed from eight weighted scoring lanes and one bonus lane, totalling a 110-point scale.
Two-container work and judge split with full architectural isolation
Multi-round delivery feedback approximating a client review cycle
Full recorded submission history making in-episode learning curves legible
Continuous scalar primary signal (not binary pass/fail)
Depth inside named professional finance verticals with public-domain US Government data
02 · Key metrics
Five numbers that define the scope and evaluation protocol of the Mephisto Samples release.
Task Bundles
5
finance RL environments
Agent Window
12h
43 200 s per task
Verifier Timeout
3 600s
separate container
Reward Scale
0–110
8 lanes + 1 bonus
Submission Cap
300
120 s cooldown
03 · Contributions
What this release establishes is structural. Five design contributions distinguish the Mephisto Samples protocol.
Contribution 01
Two-Container Work & Judge Split
Work container holds task materials and local validation tools
Judge container holds hidden evaluation assets the agent never sees
No reviewed agent benchmark documents this full architectural separation
Contribution 02
Multi-Round Delivery Feedback
Approximates a client review cycle with up to 300 submissions
120-second cooldown between submissions prevents brute-force flooding
Evaluator-only snapshots every 1 800 s capture progress the agent cannot optimize against
Contribution 03
Full Recorded Submission History
Agent-initiated and evaluator-only submissions retained in full
In-episode learning curve is recoverable after the run
Trajectory logging targets learning measurement, not just efficiency analysis
Contribution 04
Continuous Scalar Primary Signal
Per-task continuous score in [0, 1] composed from 8 weighted lanes + 1 bonus
Partial credit exists: the continuous score is the primary signal, not binary
Each lane is a linear ramp between a zero anchor and a full-marks anchor
Contribution 05
Depth in Professional Finance Verticals
Five task bundles spanning SEC XBRL fundamentals, Treasury liquidity, Fed policy regimes, leverage trajectories, and FDIC bank capital
All data sourced from US Government public domain (SEC EDGAR, FRED, FDIC BankFind, Treasury, BLS)
Anti-fabrication gate recomputes every self-reported metric from raw outputs; deviation zeroes the integrity lane plus guarded primaries
04 · Evaluation pipeline
The protocol
Two loops and a snapshot channel turn repeated graded feedback into a legible in-episode learning curve.
Phase 01
Inner Loop (local)
A writable workspace (/home/workspace) with Python, local validation, and task materials
No hidden evaluation asset is mounted; the information boundary is architectural
Unlimited fast iteration, entirely agent-driven, across the full 12 h window
Phase 02
Outer Loop (judge-mediated)
Submitted artifacts graded by a separate judge container (3 600 s timeout)
Hidden test data, expert labels, and rubric graders; both containers run no-network
Up to 300 submissions per episode, 120 s cooldown between each
Phase 03
Evaluator-Only Snapshots
Automatic grading every 1 800 s through the same hidden judge
Results retained for analysis but never shown to the agent
Measures progress the agent cannot optimize against
05 · Scoring methodology
Every bundle composes 100 base points across eight lanes plus a 10-point adaptation bonus.
Each lane is a linear ramp between a zero anchor and a full-marks anchor, so partial competence earns partial credit and the composition stays interpretable. The verifier writes a single scalar in [0, 1] normalized from the 110-point raw score.
The lane allocation below is for the FDIC Bank Capital Projection Book bundle. Other bundles follow the same 8+1 structure with domain-specific anchors.
Fig. 1: Scoring lane point allocation across the FDIC Bank Capital Projection Book.
Fig. 2a: Verifier timeout by task bundle (seconds). All bundles use 3 600 s.Fig. 2b: Submit file count by task bundle. All bundles submit 3 artifacts.
8WEIGHTED LANES100 base pts
1ADAPTATION BONUS10 bonus pts
L7ANTI-FABRICATION GATEintegrity enforced
110maximum raw score per task (100 base + 10 bonus)
06 · Dataset Viewer
Total Tasks
5
Family
1
Base Image
python
Verifier Timeout
3 600s
Loading...
Task
Family
Base Image
Submit Files
Eval Timeout
Repo
07 · Evidence status
No signed pilot evidence exists for this family at the time of release. The project disposition is HOLD:PILOT_REQUIRED. This release publishes no pass rate, no score distribution, and no model comparison.