AIZEN

FEATURE-DEVELOPMENT AGENTIC CAPABILITIES

Can AI agents implement end-to-end features across real repository context, not just isolated bug fixes?

AIZEN is a reinforcement-learning environment for training agentic coding models on complex feature development. Each instance starts from a real repository at a pinned commit, masks the implementation of a target feature while preserving enough interface and behavioural context for the task to be executable, and grades the agent by running tests rather than by judging prose. Reward is deterministic and execution-derived: the requested feature must move from failing to passing while protected surrounding behaviour stays passing. Tasks are generated from repository scanning, dynamic tracing, masking, and post-verification — not hand-authored — and each bundle carries the provenance needed to reconstruct where it came from and which verifier contract grades it.

  • Execution-verified reward: fail-to-pass and pass-to-pass test contracts computed from verifier output — no LLM judge, no reference transcript, no prompt-parroting credit.
  • Reproducible containerized substrate: pinned repository commit, Docker environment metadata, and an explicit callable interface contract per task.
  • Feature-level long-horizon work: L1 tasks preserve repository context for incremental feature work; L2 tasks require from-scratch implementation against the interface contract.

Tasks

10,000

target task environments

Domains

7

LLM infra · ML frameworks · scientific computing · data tooling · web/API · type & test infra · build & workflow

Languages

5

Python · Go · Rust · JavaScript · TypeScript

Principle 01

Execution Over Narration

  • Reward formula: pass_rate = success / total, computed from the verifier's test status map.
  • Binary resolution: is_solved is 1 only when every FAIL_TO_PASS and every PASS_TO_PASS test passes.
  • Partial credit is the training gradient; binary resolution is the ship bar.
  • Reward derives from verifier output, never from a model's explanation of what it changed.

Principle 02

Retention Gates

A candidate that misses any gate is dropped, not downgraded.

  • Pending implementation lines — > 100
  • FAIL_TO_PASS test points — ≥ 10
  • Callable interface — explicit, specified
  • Source test commit date — after May 2022
  • Source URLs — no blocked or blacklisted entries
  • Environment — reproducible Docker metadata

Principle 03

Provenance Binding

Every emitted bundle carries: pinned source repository and commit · generator and harness commits · Harbor schema version · environment image or build recipe · fail-to-pass and pass-to-pass verifier contract · source licence and attribution obligations · difficulty level (L1/L2) and callable interface spec · any blocked URLs.

The method

Three phases from real repository to scored result.

Phase 01

Discover & Trace

  • Clone the source repository at a pinned commit; build the base and instance environment images.
  • Discover fail-to-pass and pass-to-pass candidates from the repository's own test suite.
  • Verify candidate pass-to-pass files run 100% green on the unmodified repository.
  • Dynamic-trace test execution into a function-level dependency graph.

Phase 02

Classify & Mask

  • Classify which imported objects are the feature target versus test utilities.
  • Draw the pass-to-pass guard set that must survive masking.
  • Mask the implementation via breadth-first traversal of the dependency graph, leaving intact what the guard set requires.
  • Post-verify deterministically: the stripped repository must fail every FAIL_TO_PASS test and pass every PASS_TO_PASS test, and the reference patch must make the complete suite pass. Bad candidates are dropped.

Phase 03

Render & Certify

  • Render problem_statement at two levels: L1 (interface context present) and L2 (from scratch).
  • Emit the self-contained Harbor bundle: task.toml, instruction.md, environment/, tests/.
  • Certify: the reference solution must score 1.0 and a no-op submission must score 0.0. A bundle that fails either does not ship.

Fig. 1: Per-domain resolution rate across evaluated models.

Fig. 2: %PASSED versus %RESOLVED by difficulty level (L1, L2).
Fig. 3: Failure mode distribution.
Loading...
Instance Domain Difficulty Task Golden End-State Model A Model B

Head-to-head breakdown of both evaluated frontier models.

anthropic/claude-opus-4-8

Overall accuracy

openai/gpt-5.6-sol

Overall accuracy