Phase 01
Data Curation
- Collect real-world documents: PDFs, images, videos, web pages
- Design multi-step questions requiring cross-modal reasoning
- Establish deterministic ground truth answers
- Assign difficulty levels (Level 1, Level 2, Level 3)
§01 · Overview
TERRA is an RL environment that trains AI assistants on multi-modal reasoning and tool-use tasks following the GAIA methodology. Given real-world documents, images, videos, and web-accessible information, agents must decompose complex questions into multi-step reasoning chains, select appropriate tools, and produce precise answers. Terra requires coordinating across PDF analysis, web search, image understanding, video comprehension, and mathematical reasoning. This mirrors the complexity of real-world information work.
- Multi-modal tasks across 5 categories: Web Browsing, File Reading, Multi-Modality, Calculation, Self-Contained
- Tool-use challenges: PDF reader, Web search, Calculator, Image/Video analysis
- Three difficulty levels: Hard, Very Hard, Expert
- Exact-match scoring with deterministic ground truth answers
- GAIA methodology for reproducible multi-step training
§02 · Key metrics
Three numbers that define the scope of Terra.
Instances Curated
10,000
across 5 categories
Categories
5
calculation, multimodality, file reading, self-contained, web browsing
Difficulty Levels
3
Hard · Very Hard · Expert
§03 · Methodology
Three scoring principles ensure fair and reproducible results.
01
Exact Match
Answers must match ground truth exactly. No partial credit, no fuzzy matching.
02
Tool Transparency
All tool calls are logged. Models must demonstrate appropriate tool selection.
03
Reproducibility
Fixed seeds, deterministic scoring, published solution traces for independent verification.
§04 · Evaluation pipeline
The method
Three phases turn raw documents into a training environment.
Phase 02
Agent Execution
- Deploy AI agents with access to tools (PDF reader, web search, calculator)
- Agents decompose questions into multi-step reasoning chains
- Track tool selection and execution traces
- Record complete interaction traces for analysis
Phase 03
Reward Signal
- Compare agent answers against ground truth via exact match
- Binary reward: correct or incorrect, no partial credit
- Compute per-model accuracy across difficulty levels
- Generate per-category and per-difficulty breakdowns
§05 · Results
We validated difficulty calibration using two frontier models on the dataset.
Results computed dynamically from the dataset below.
Kimi K2.5
-
Loading...
Qwen3-VL
-
Loading...
Hard
-
-
-
Kimi K2.5
-
-
Qwen3-VL
-
-
Combined
Very Hard
-
-
-
Kimi K2.5
-
-
Qwen3-VL
-
-
Combined
Expert
-
-
-
Kimi K2.5
-
-
Qwen3-VL
-
-
Combined
§06 · Dataset Viewer
Hard
-
-
Kimi K2.5
-
Qwen3-VL
-
Failure Rate
Very Hard
-
-
Kimi K2.5
-
Qwen3-VL
-
Failure Rate
Expert
-
-
Kimi K2.5
-
Qwen3-VL
-
Failure Rate
| Task ID | Difficulty | Category | Kimi K2.5 | Qwen3-VL |
|---|
§07 · Model comparison
Head-to-head breakdown of both evaluated frontier models.
Kimi K2.5
- Overall Accuracy
- 50% (10/20)
- Hard
- 100% (4/4)
- Very Hard
- 85.7% (6/7)
- Expert
- 0% (0/9)
- Avg LLM Calls
- 16.8
- Avg Latency
- 3.6s
Qwen3-VL
- Overall Accuracy
- 10% (2/20)
- Hard
- 25% (1/4)
- Very Hard
- 14.3% (1/7)
- Expert
- 0% (0/9)
- Avg LLM Calls
- 21.9
- Avg Latency
- 10.3s
§08 · Resources