RAIDEN

AGENTIC CODING · RL ENVIRONMENT

Can a coding agent build a stateful command-line tool from a spec it is never tested against?

by Ethara AI

GitHub Hugging Face Get in touch

01 · Overview

Raiden is a reinforcement-learning environment, not a static Q/A set, for agentic code generation. This sample spans 30 tasks: a Kubernetes CLI (kubectl against a kwok control plane) plus six AWS service families (DynamoDB, KMS, SQS, Cognito, S3, Kinesis). Each drops a coding agent into a container with a prose spec and no test code, asks it to build the CLI surface from scratch, and grades the result against a hidden end-to-end suite. The pass fraction is the score. Every surface is deliberately stateful: resources persist across commands, so the task rewards genuine implementation over recall. Each backend is a hermetic, offline sidecar (kwok, DynamoDB Local, local-kms, ElasticMQ, cognito-local, MinIO, kinesalite).

  • Held-out grading: the agent never sees the 6,852 E2E tests
  • Seven CLI surfaces: Kubernetes (kubectl) + six AWS families
  • Dense scalar score = passed / total ∈ [0, 1], anti-NOP guarded
  • Hermetic, offline simulation sidecars per family
  • Two-model calibration: Claude Opus 4.8 and GPT-5.6-sol

02 · Key metrics

Three numbers that define the scope of the Raiden delivery, across its seven CLI surfaces.

Harbor Tasks

30

10 kubernetes · 20 aws

Held-out Tests

6,852

across all seven families

Calibration Models

2

Opus 4.8 · GPT-5.6-sol

03 · Methodology

How the reward is computed. Four principles govern the Raiden grading system.

Principle 01

Held-Out Grading

  • The agent never sees any test code during generation
  • It must satisfy real service semantics, not pattern-match the grader
  • Graded on observable behaviour: stdout, stderr, exit code, resource state

Principle 02

Dense Scalar Reward

  • reward = passed / total ∈ [0, 1], the simplest dense signal
  • No partial credit within a test, no weighting, no exit-code shortcut
  • Turns the suite into a dense curriculum over CLI behaviours

Principle 03

Anti-NOP Floor & Anti-Gaming

  • A conftest guard aborts before collection if submission/aws is absent
  • An empty / missing / non-executable stub scores exactly 0.0
  • Closed exit-code contract: {0, 1, 252, 254, 255}

Principle 04

Hermetic & Offline

  • Per-family sidecars (kwok, DynamoDB Local, local-kms, and more), sha256-pinned, offline
  • A socket-connect guard blocks any non-loopback / non-private connect
  • PYTHONHASHSEED=0 · TZ=UTC · LC_ALL=C.UTF-8 baked into the image

04 · Evaluation pipeline

The method

Three phases turn a prose spec into a scored, reproducible reinforcement-learning signal.

Phase 01

Environment Setup

  • Build the task container from a pinned sha256 base image
  • Ship an instruction.md spec: commands, I/O, error cases
  • Boot the per-family simulation sidecar: kwok, DynamoDB Local, local-kms, ElasticMQ, cognito-local, MinIO, kinesalite
  • Put /workspace/submission/ first on PATH

Phase 02

Agent Interaction

  • The agent writes an aws executable from scratch (any language)
  • It dispatches every subcommand in its subset (kubectl or aws)
  • State stays consistent across a multi-command workflow
  • The agent never sees the held-out tests

Phase 03

Evaluation & Scoring

  • Apply the submission and run pytest with randomization off
  • Cross-check collected count against tests_shipped
  • Compute reward = passed / total and write reward.txt
  • Always exit 0; the reward file is the grading channel

05 · Results

Two frontier models were run end-to-end against all 30 tasks under the openhands-sdk / harbor scaffold (pass@1): Claude Opus 4.8 and GPT-5.6-sol, for 60 graded runs.

Across the corpus GPT-5.6-sol reaches a mean score of 0.8932 and Opus 4.8 0.7814. Binning each model's own 30 tasks into five score bands (Trivial S≥.95 to Expert S<.50) gives a clean staircase decay for both, and effort per task, in dollars and output tokens, climbs into the harder tiers.

Mean score per per-model difficulty band, and the effort each spends per task in dollars and output tokens. All three figures are computed live from each run's verifier score, cost, and token accounting.

Fig. 1: Score decay across difficulty tiers. Mean score per per-model score band; both saturate the top bands, and GPT-5.6-sol holds the Expert tail markedly better than Opus 4.8.
Fig. 2a: Cost escalation. Mean cost per task (USD); GPT-5.6-sol spends more dollars per task than Opus in every shared tier.
Fig. 2b: Token bloat. Mean output tokens per task; Opus 4.8 emits more tokens, a pricing-versus-verbosity split.
12 TRIVIAL / EASY 40.0%
6 MEDIUM 20.0%
12 HARD / EXPERT 40.0%
89.3% GPT-5.6 Mean Score

06 · Dataset Viewer

Total Tasks

30

Scopes

7

Held-out Tests

6,852

Opus Mean Score

78.1%

Loading...
Task Commands Difficulty Opus 4.8 GPT-5.6-sol Tests

07 · Scope comparison

The two largest scoped surfaces side by side. On Kubernetes the two models track closely, while on DynamoDB GPT-5.6 pulls well ahead of Opus 4.8.

kubernetes kubectl / 10 tasks

Opus 4.8 mean
78.4%
GPT-5.6 mean
78.6%
Mean gap
+0.2 pts
Backend
kwok

aws dynamodb / 8 tasks

Opus 4.8 mean
60.3%
GPT-5.6 mean
99.6%
Mean gap
+39.3 pts GPT-5.6
Backend
DynamoDB Local

08 · Resources

GitHub

raiden-samples

github.com/EtharaOrion/raiden-samples

Hugging Face

raiden-samples

huggingface.co/datasets/ethara/raiden-samples

Need a custom RL environment or eval harness?

Ethara builds held-out, reproducible agent environments and datasets for frontier teams.

Get in touch