Zoro is an Ethara.AI initiative that builds a structured evaluation benchmark on top of RetailBench, a simulation environment where an AI agent manages a single-store supermarket over 180 simulated days. The benchmark turns the open-ended RetailBench simulator into a rigorous, reproducible task suite that measures how well frontier language models handle complex, multi-step business decisions over time. Running a store is not a single decision — it requires coordinating inventory, pricing, supplier selection, customer reviews, news events, and cash position simultaneously, where a bad stocking decision in week one can cascade into bankruptcy by week eight. The oracle policy — with privileged access to supplier quality scores — achieves $131,510 net worth and 267,998 units sold. GPT-5.5, the strongest model tested, reaches just 18.6% of that ceiling.
Long-horizon planning: decisions made on day 5 directly affect outcomes on day 80
Evidence-grounded reasoning: product quality data must be gathered before acting, not assumed
Consistent strategy execution: the same decision logic applied repeatedly across 180 cycles
Temporal follow-up: orders placed must be tracked; quality issues discovered must trigger corrective action
Oracle-verified difficulty: only achievable tasks enter the benchmark — trivial and broken configs filtered out
02 · Key metrics
Three numbers that define the scope of the Zoro evaluation harness, grounded in the RetailBench simulator.
Target Evaluation Tasks
10,000
2k hard · 6k middle · 2k easy
Simulation Horizon
180
days per evaluation task
Oracle Net Worth
$131,510
reference ceiling (not a target)
03 · Methodology
Four components define Zoro's evaluation framework, from task construction to failure attribution.
Component 01
Task Suite Design
~10,000 versioned tasks, each pinning all 27 RetailBench environment parameters
Four presets × five demand elasticity profiles × five economic tiers × three news intensity levels
Tasks stored in a stable dataset.json schema with preset, scenario, seed, and oracle verdict
Component 02
Oracle Verification
Oracle policy runs on every candidate task with privileged access to supplier quality scores
Tasks where oracle pass_threshold_ratio is in [0.2, 0.8] are marked achievable and enter the benchmark
Trivial (> 0.8) and impossible (< 0.2) configurations are excluded automatically
Component 03
Evaluation Harness
Five agent scaffolds: ReAct, Plan-and-Act, Reflection, Step-Reflection, Exec-Strategy
Tool calls communicated via XML parsing — works identically across any model regardless of native function-calling support
Every tool call logged to tool_calls.jsonl across all 180 days
Each metric maps to one of three identified failure modes: incomplete evidence, surface-level decisions, inconsistent policy
Computed post-hoc from tool_calls.jsonl — not from simulation state
04 · Evaluation pipeline
The method
Three phases turn an environment configuration into a scored, reproducible reinforcement-learning signal.
Phase 01
Instance Preparation
Assign a preset, scenario overlay, and random seed to pin all 27 environment parameters
Run the oracle policy to verify the task is achievable — not trivially easy, not structurally broken
Record the oracle net worth as the reference ceiling for that task
Phase 02
Agent Evaluation
Run the agent under one of five scaffolds across all 180 simulated days
Agent reads state via read-only tools, then issues action tools to order, price, and manage listings
Every tool call and response is logged to tool_calls.jsonl
Phase 03
Scoring & Diagnostics
Primary metric: net worth at day 180 expressed as % of oracle ceiling
Secondary: eight diagnostic metrics computed from tool_calls.jsonl to attribute failures
Multiple seeds run per task to estimate variance from stochastic demand
05 · Results
Seven frontier models were benchmarked on the dynamic_hard preset under the ReAct scaffold across 180 simulated days (results from the RetailBench paper; arXiv:2603.16453).
GPT-5.5 achieved 18.6% of the oracle's $131,510 net worth — the strongest result of seven models tested. DeepSeek-V4-Pro reached 7.7%. Five of seven models bankrupted before day 180, ending with negative net worth.
Fig. 1: Estimated mean GPT-5.5 score (% of oracle) by demand elasticity profile. Inelastic demand tasks yield higher scores; mixed-demand tasks present the greatest challenge.
Score by environment preset, and planned task distribution across presets.
Fig. 2: Estimated GPT-5.5 mean score by environment preset. dynamic_hard is the most challenging; still_middle removes news and customer reviews for a cleaner inventory signal.
Fig. 3: Planned task distribution by environment preset, stacked by demand elasticity type. 10,000 total tasks across four presets.
1ABOVE 50% ORACLE6.3%
12PARTIAL 1–49%75.0%
3BANKRUPT / UNGRADED18.8%
18.6%GPT-5.5 Best Score (% of Oracle, dynamic_hard)
06 · Dataset Viewer
Dataset Scenarios
—
Presets
—
SKUs (hard preset)
—
GPT-5.5 Mean Score
—
Loading...
Scenario
Preset
Difficulty
GPT-5.5
SKUs
07 · Environment presets
Four presets define Zoro's difficulty landscape. Dynamic presets activate news events and customer reviews, introducing information asymmetry. Still presets use static pricing for a cleaner inventory signal.