RAIDEN

AGENTIC CODING · RL ENVIRONMENT

Can a coding agent build a stateful command-line tool from a spec it is never tested against?

by Ethara AI

GitHub Hugging Face Get in touch

01 · Overview

Raiden is a reinforcement-learning environment, not a static Q/A set, for agentic code generation. It ships two scoped deliveries: 10 aws s3 tasks and 10 aws dynamodb tasks. Each drops a coding agent into a container with a prose spec and no test code, asks it to build a subset of the CLI surface from scratch, and grades the result against a hidden end-to-end suite. The pass fraction is the reward. Both surfaces are deliberately stateful: buckets/objects and tables/items persist across commands, so the task rewards genuine implementation over recall. S3 is simulated by a per-session MinIO server and DynamoDB by an amazon/dynamodb-local sidecar, offline by construction.

  • Held-out grading: the agent never sees the 1,577 E2E tests
  • Two CLI surfaces: 7 aws s3 + 8 aws dynamodb commands
  • Dense scalar reward = passed / total ∈ [0, 1], anti-NOP guarded
  • Hermetic, offline MinIO (S3) and DynamoDB Local (DynamoDB) sidecars
  • Two-model calibration: Claude Opus 4.8 and Claude Haiku 4.5

02 · Key metrics

Three numbers that define the scope of the Raiden delivery, across its two CLI surfaces.

Harbor Tasks

20

10 aws s3 · 10 aws dynamodb

Held-out Tests

1,577

626 aws s3 · 951 aws dynamodb

Calibration Models

2

Opus 4.8 · Haiku 4.5

03 · Methodology

How the reward is computed. Four principles govern the Raiden grading system.

Principle 01

Held-Out Grading

  • The agent never sees any test code during generation
  • It must satisfy real DynamoDB semantics, not pattern-match the grader
  • Graded on observable behaviour: stdout, stderr, exit code, table/item state

Principle 02

Dense Scalar Reward

  • reward = passed / total ∈ [0, 1], the simplest dense signal
  • No partial credit within a test, no weighting, no exit-code shortcut
  • Turns the suite into a dense curriculum over DynamoDB behaviours

Principle 03

Anti-NOP Floor & Anti-Gaming

  • A conftest guard aborts before collection if submission/aws is absent
  • An empty / missing / non-executable stub scores exactly 0.0
  • Closed exit-code contract: {0, 1, 252, 254, 255}

Principle 04

Hermetic & Offline

  • MinIO (S3) and DynamoDB Local (DynamoDB) sidecars, sha256-pinned, offline
  • A socket-connect guard blocks any non-loopback / non-private connect
  • PYTHONHASHSEED=0 · TZ=UTC · LC_ALL=C.UTF-8 baked into the image

04 · Evaluation pipeline

The method

Three phases turn a prose spec into a scored, reproducible reinforcement-learning signal.

Phase 01

Environment Setup

  • Build the task container from a pinned sha256 base image
  • Ship an instruction.md spec: commands, I/O, error cases
  • Boot the simulation sidecar: MinIO for S3, DynamoDB Local for DynamoDB
  • Put /workspace/submission/ first on PATH

Phase 02

Agent Interaction

  • The agent writes an aws executable from scratch (any language)
  • It dispatches every s3 / dynamodb subcommand in its subset
  • State stays consistent across a multi-command workflow
  • The agent never sees the held-out tests

Phase 03

Evaluation & Scoring

  • Apply the submission and run pytest with randomization off
  • Cross-check collected count against tests_shipped
  • Compute reward = passed / total and write reward.txt
  • Always exit 0; the reward file is the grading channel

05 · Results

Two Anthropic models were run end-to-end against all 20 tasks under the openhands-sdk v1.12.0 / harbor 0.13.1 scaffold (pass@1).

Measured as a share of Opus 4.8, Haiku 4.5 captures nearly all of Opus's reward on the easiest tasks (98%) but that share decays steadily toward 0% on the hardest tail. Aggregated across all 20 tasks Haiku recovers 51% of Opus's reward (48.9% vs 95.6% mean).

Reward per dollar per task, each model ranked by its own cost-efficiency (best to worst): efficiency decays for both models, and even Opus 4.8 falls by two thirds on its worst tasks.

Fig. 1: Cost-efficiency decay. Reward per USD per task, ranked best to worst per model. Opus 4.8 falls 66% from its best task to its worst; Haiku 4.5 falls 100%.
Fig. 2a: S3 tasks. Opus 4.8 vs Haiku 4.5 per task (dumbbell = capability gap), sorted by Haiku reward.
Fig. 2b: DynamoDB tasks. Opus 4.8 vs Haiku 4.5 per task (dumbbell). Haiku spans 0% to 98%; Opus holds ≥94%. Longer dumbbell = more RL headroom.
6 EASY · HAIKU 75%+ 30.0%
4 MEDIUM · 50-75% 20.0%
10 HARD · < 50% 50.0%
51.1% Haiku 4.5 Share of Opus Reward

06 · Dataset Viewer

Total Tasks

20

Scopes

2

Held-out Tests

1,577

Opus Mean Reward

95.6%

Loading...
Task Commands Difficulty Opus 4.8 Haiku 4.5 Tests

07 · Scope comparison

The two scoped deliveries side by side. Opus 4.8 stays near the ceiling on both surfaces while Haiku 4.5 lands well below, widest on the S3 surface.

aws s3 · 10 tasks

Opus 4.8 mean
92.8%
Haiku 4.5 mean
44.8%
Mean gap
+48.0 pts
Held-out tests
626
Backend
MinIO

aws dynamodb · 10 tasks

Opus 4.8 mean
98.4%
Haiku 4.5 mean
53.0%
Mean gap
+45.4 pts
Held-out tests
951
Backend
DynamoDB Local

08 · Resources

GitHub

raiden-samples

github.com/EtharaOrion/raiden-samples

Hugging Face

raiden-samples

huggingface.co/datasets/ethara/raiden-samples

Need a custom RL environment or eval harness?

Ethara builds held-out, reproducible agent environments and datasets for frontier teams.

Get in touch