KANG · Financial-Reasoning Benchmark

01 · Overview

Kang is a reinforcement-learning environment for training and evaluating agents on hard, open-ended financial-analyst reasoning. Each task drops an agent into a sandboxed container stocked with a point-in-time corpus of primary documents (SEC, NSE, BSE, etc filings, earnings calls, financials, press), hands it an open-ended research topic, and scores the written analysis against an explicit rubric of the analytical moves a competent analyst would make. Everyone is building finance agents; no one can measure their reasoning. Existing benchmarks terminate in checkable answers and grade factual recall; Kang grades the argument itself: conviction, rigour, synthesis, and grounding. The figures on this page come from the public kang-samples release: 30 environments and two recorded baselines, claude-opus-4-8 and gpt-5.6-sol, graded cross-family with the judges swapped.

Realistic: on-the-job tasks from working hedge-fund analysts
Deterministic: open-ended reasoning graded against verified expert moves
Reasoning-first: argument and judgement, not recall or retrieval
Uncontaminated: proprietary traces no model saw in pretraining
Dense 0-4 rubric scored on grounding, coverage, and synthesis

02 · Key metrics

Three numbers that define the scope of this public sample run.

Sample Environments

30

4 themes each · 375 graded moves

Evaluation Trials

60

2 agents · 1 trial each

pass@1

28/60

0/30 and 28/30, different judges

03 · Methodology

What earns credit. Four requirements govern every graded answer.

Requirement 01

Take a Position · Conviction

Commit to a clear view rather than surveying both sides
A hedged non-answer earns no credit
Judgement is the skill being measured

Requirement 02

Engage Counter-Evidence · Rigour

Confront the strongest opposing data in the corpus
Not just the evidence that supports the thesis
Graded on coverage of the required moves

Requirement 03

Reconcile Conflicts · Synthesis

Fuse opposing data points into one unified conclusion
An explicit reconciliation sentence is required
Reserved for the top score of 4.0

Requirement 04

Cite Every Claim · Grounding

Every factual claim cites its source file inline
Ungrounded facts are discarded by the grader
A move resting on a flagged fact is forfeited

04 · Grading pipeline

The judge

A three-check judge turns an open-ended answer into a deterministic 0-4 score.

The judge is not fixed by the benchmark: each run records the model that scored it. Both shipped baselines were graded cross-family to avoid self-grading bias, which means the two arms do not share a judge.

Check 01

Grounding

Extract every factual claim from the answer
Verify each against its cited source; flag the unverifiable
A move resting on a flagged fact is forfeited

Check 02

Coverage

Per theme, which lettered required moves are hit
Matched by concept, not vocabulary
A theme is covered when grounded moves reach its threshold

Check 03

Synthesis

One explicit reconciliation of opposing data
Unlocks the top score of 4.0
Covering 3 of 4 themes scores the same as 2: the top tiers need all four
Pass = a perfect 4.0 / 4.0

05 · Results

The public sample release ships two recorded baselines over the same 30 environments, one trial each: 60 evaluation trials in total.

The two arms were graded by different judges, swapped, so neither run is self-graded: claude-opus-4-8 was scored by gpt-5.6-sol, and gpt-5.6-sol by claude-opus-4-8. Under that setup claude-opus-4-8 records a mean dense score of 0.33 / 4.0 and pass@1 of 0/30, while gpt-5.6-sol records 3.87 / 4.0 and 28/30. The entire gap runs through the grounding pass: synthesis was found in all 60 runs, so no trial has yet scored 3, but 78.7% of the moves credited to claude-opus-4-8 were tainted against 4.7% for gpt-5.6-sol.

Fig. 1: What concept matching credits, against what survives grounding. A grader that stops at concept match scores claude-opus-4-8 at 61.3% of the rubric; Kang's grounding pass then discards 181 of those 230 moves because the cited filings do not support them, leaving 13.1%. The same pass costs gpt-5.6-sol 4.5 points. That 48.3-point drop is the channel finance benchmarks without a grounding check never apply.

Difficulty is graded, not uniform. Environments ranked by grounded-move rate and cut into five tiers of six.

Fig. 2: Decay across difficulty tiers. The 30 environments are ranked by combined grounded-move rate and cut into five tiers of six, so the combined line falls by construction; what the tiers reveal is the shape of each arm. claude-opus-4-8 decays monotonically, 0.33 down to 0.03. gpt-5.6-sol holds near its ceiling through Hard and only gives way at Expert, 0.93 to 0.82 - the top sixth is where headroom remains even for a saturated arm.

Fig. 3: Per-environment calibration band. Every environment plotted once per agent, ranked easiest to hardest, against a golden 1.0 where every rubric move is landed and grounded. The spread runs from DAL at 0.67 to CAT at 0.31 combined; no environment in the sample is solved outright by both arms, and none is unreachable by both.

0.33 MEAN DENSE - CLAUDE-OPUS-4-8 judged by gpt-5.6-sol

3.87 MEAN DENSE - GPT-5.6-SOL judged by claude-opus-4-8

60/60 SYNTHESIS FOUND no trial has scored 3

78.7% vs 4.7% share of credited moves tainted - claude-opus-4-8 vs gpt-5.6-sol, different judges

06 · Trial Leaderboard

Environments

30

Agents

2

Trials

60

pass@1

28/60

Loading...

Environment	Agent / Judge	Sector	Dense 0-4	Themes	Moves grounded

07 · Where Kang sits

Existing finance benchmarks grade checkable answers or process. Kang is the one grading open-ended reasoning deterministically.

The landscape

FinQA / TAT-QA: checkable answers
FinanceBench: factual recall
FAB v2: process-graded
Kang: open-ended reasoning

The moat

Analyst network: perpetual IP licences
Grading: deterministic, open-ended
Tasks: uncontaminated
Public sample: 30 environments · 2 baselines

08 · Resources

GitHub

kang-samples

github.com/EtharaOrion/kang-samples

Hugging Face

kang

huggingface.co/datasets/ethara/kang-samples

Need a hard reasoning eval or expert-aligned preference data?

Ethara builds held-out, reproducible agent benchmarks and datasets for frontier model labs and financial institutions.

Get in touch