AIZEN is a reinforcement-learning environment for training agentic coding models on complex feature development. Each instance starts from a real repository at a pinned commit, masks the implementation of a target feature while preserving enough interface and behavioural context for the task to be executable, and grades the agent by running tests rather than by judging prose. Reward is deterministic and execution-derived: the requested feature must move from failing to passing while protected surrounding behaviour stays passing. Tasks are generated from repository scanning, dynamic tracing, masking, and post-verification — not hand-authored — and each bundle carries the provenance needed to reconstruct where it came from and which verifier contract grades it.
Execution-verified reward: fail-to-pass and pass-to-pass test contracts computed from verifier output — no LLM judge, no reference transcript, no prompt-parroting credit.
Reproducible containerized substrate: pinned repository commit, Docker environment metadata, and an explicit callable interface contract per task.
Feature-level long-horizon work:L1 tasks preserve repository context for incremental feature work; L2 tasks require from-scratch implementation against the interface contract.
02 · Key metrics
Tasks
10,000
target task environments
Domains
7
LLM infra · ML frameworks · scientific computing · data tooling · web/API · type & test infra · build & workflow
Languages
5
Python · Go · Rust · JavaScript · TypeScript
03 · Methodology
Principle 01
Execution Over Narration
Reward formula: pass_rate = success / total, computed from the verifier's test status map.
Binary resolution: is_solved is 1 only when every FAIL_TO_PASSand every PASS_TO_PASS test passes.
Partial credit is the training gradient; binary resolution is the ship bar.
Reward derives from verifier output, never from a model's explanation of what it changed.
Principle 02
Retention Gates
A candidate that misses any gate is dropped, not downgraded.
Pending implementation lines — > 100
FAIL_TO_PASS test points — ≥ 10
Callable interface — explicit, specified
Source test commit date — after May 2022
Source URLs — no blocked or blacklisted entries
Environment — reproducible Docker metadata
Principle 03
Provenance Binding
Every emitted bundle carries: pinned source repository and commit · generator and harness commits · Harbor schema version · environment image or build recipe · fail-to-pass and pass-to-pass verifier contract · source licence and attribution obligations · difficulty level (L1/L2) and callable interface spec · any blocked URLs.
04 · Evaluation pipeline
The method
Three phases from real repository to scored result.
Phase 01
Discover & Trace
Clone the source repository at a pinned commit; build the base and instance environment images.
Discover fail-to-pass and pass-to-pass candidates from the repository's own test suite.
Verify candidate pass-to-pass files run 100% green on the unmodified repository.
Dynamic-trace test execution into a function-level dependency graph.
Phase 02
Classify & Mask
Classify which imported objects are the feature target versus test utilities.
Draw the pass-to-pass guard set that must survive masking.
Mask the implementation via breadth-first traversal of the dependency graph, leaving intact what the guard set requires.
Post-verify deterministically: the stripped repository must fail every FAIL_TO_PASS test and pass every PASS_TO_PASS test, and the reference patch must make the complete suite pass. Bad candidates are dropped.
Phase 03
Render & Certify
Render problem_statement at two levels: L1 (interface context present) and L2 (from scratch).
Emit the self-contained Harbor bundle: task.toml, instruction.md, environment/, tests/.
Certify: the reference solution must score 1.0 and a no-op submission must score 0.0. A bundle that fails either does not ship.
05 · Results
—
—
—
Fig. 1: Per-domain resolution rate across evaluated models.
—
—
Fig. 2: %PASSED versus %RESOLVED by difficulty level (L1, L2).
—
Fig. 3: Failure mode distribution.
———
———
———
——
06 · Dataset Viewer
Loading...
Instance
Domain
Difficulty
Task
Golden End-State
Model A
Model B
07 · Model comparison
Head-to-head breakdown of both evaluated frontier models.