Principle 01
Specification-Driven Generation
- Each instance provides a specification and build config.
- The model must implement the stubbed library: source, build files
- No partial credit: the library must compile and pass its held-out suite
01 · Overview
Kaiju is a reinforcement-learning environment, not a static Q/A set, for agentic code generation. It ships from-scratch library-generation tasks across Go, Python, C++, C, TypeScript, JavaScript, Java, and Rust. Each drops a coding agent into a hermetic container with a specification and build config and no test code, asks it to implement a complete open-source stubbed library from scratch, and grades the result against a held-out test suite. The module-level pass fraction is the reward. Every task is drawn from a real open-source project, so the environment rewards genuine implementation over recall. Builds run in isolated, offline containers with a per-project test command (pytest, go test, cargo test, etc.).
- Held-out grading: the agent never sees the tests it is scored on
- Three languages: Go, Python, and Rust libraries built from scratch
- Dense scalar reward = modules passed / total ∈ [0, 1], reported per stage
- Hermetic, offline build verification with a per-project test harness
- Single-model calibration: Claude Opus 4.8 across a three-stage (draft, lint, test) pipeline
02 · Key metrics
Three numbers that define the scope of the Kaiju delivery, across three languages.
Library Tasks
16
5 Go · 2 Python · 9 Rust
Average Modules Modified
32
across 20 libraries
Calibration Model
1
Claude Opus 4.8
03 · Methodology
How the reward is computed. Four principles govern the Kaiju grading system.
Principle 02
Three-Stage Pipeline
- Stage 1 Draft: initial generation from the specification
- Stage 2 Lint: iterative refinement to pass static analysis
- Stage 3 Test: iterative refinement to pass the test suite
Principle 03
Module-Level Pass Rate
- Reward = modules passed / total ∈ [0, 1], measured at module granularity
- A module passes only when ALL its tests succeed
- Reported per stage and per language
Principle 04
Hermetic Build Verification
- Each instance runs in an isolated, offline container
- A per-project test command drives grading (pytest, go test, cargo test)
- Deterministic pass/fail with no environment variability
04 · Evaluation pipeline
The method
Three phases turn a specification into a scored, reproducible reinforcement-learning signal.
Phase 01
Instance Preparation
- Identify real open-source libraries across Go, Python, and Rust
- Capture the specification and build configuration
- Pin base and reference commits for reproducibility
Phase 02
Code Generation
- Run the agent in a sandbox from the specification alone
- Generate the complete library implementation from stubbed code
- Produce per-module outputs across the three stages
Phase 03
Evaluation & Scoring
- Run the held-out suite with the per-project test command
- Compute reward = modules passed / total
- Report the pass rate per stage and per language
05 · Results
Claude Opus 4.8 was run end-to-end against all 16 tasks (pass@1).
Reward spans the full range: from 0% on several Rust and Go libraries to 100% on jsonrpc-core, with a graded mean of 29.7% across the 10 scored instances. Python leads (48.0% mean) ahead of Rust (25.1%); the five Go tasks remain ungraded.
Pass rate by language, and task distribution across languages.
3
SOLVED ≥ 50%
18.8%
3
PARTIAL 1–49%
18.8%
10
UNSOLVED / UNGRADED
62.5%
29.7%
Claude Opus 4.8 Mean Reward (10 graded)
06 · Dataset Viewer
Total Tasks
16
Languages
3
Average Modules Modified
32
Opus Mean Reward
29.7%
| Task | Language | Difficulty | Opus 4.8 | Tests |
|---|
07 · Language comparison
The three languages side by side. Python leads on reward; Rust is the largest slice and runs longest; the Go tasks remain ungraded.
08 · Resources
GitHub
kaiju-samples
github.com/EtharaOrion/kaiju-samples
Hugging Face
kaiju-samples
huggingface.co/datasets/ethara/kaiju-samples
Need a custom coding benchmark or eval harness?
Ethara builds held-out, reproducible agent environments and datasets for frontier teams.