RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers
Haitong Ma, Chenxiao Gao, Rushi Qiang, Bo Dai, Na Li
cs.RO
2026-09-28
RLE-Bench runs 11 frontier coding agents through 51 physics-simulation robot-engineering tasks across four workflows; GPT-6 Astra leads with an RLE Index of 73.4.
Coding agents can already read unfamiliar repositories, run code, interpret failing tests, and revise their work over long sessions; SWE-bench and Terminal-Bench measure exactly that. Robotics benchmarks measure something else. LIBERO, RoboTwin, and their relatives score trained policies and controllers, not the engineering work that surrounds them. Real robotics development extends well past control: someone has to design the mechanical structure, write the perception stack, train the policy, and debug failures inside a time and compute budget. CaP-X took a step in this direction by testing whether agents can write controller code, but it stops at code-as-policy. RLE-Bench, from Harvard and Georgia Tech, makes the coding agent itself the unit under test and asks whether frontier models can carry digital engineering skill into physics-grounded work.
51 tasks in 9 families, grouped into four workflows. Formally each task is a tuple: instruction, public development environment, resource budget (wall clock, interaction steps, compute), an artifact contract, a hidden evaluation environment, and a verifier. The agent builds under budget and submits an executable artifact; the verifier scores it through physical rollouts under hidden seeds, scenes, embodiments, or held-out tasks, with privileged state kept verifier-private. Everything ships as Harbor container environments. The headline RLE Index is an equally weighted macro-average of the four workflow scores.\n
Eleven model-harness combinations, all run at high reasoning effort with web search disabled.
| System (harness) | RLE Index | Mean cost/task |
| GPT-6 Astra (Codex CLI) | 73.4 | $39.00 |
| Claude Fable 5.1 (Claude Code) | 64.0 | $22.81 |
| Claude Opus 5 (Claude Code) | 53.4 | $27.01 |
| GPT-5.6 Sol (Codex CLI) | 48.7 | $17.17 |
| DeepSeek-V4.1-Flash (Claude Code) | 38.9 | $0.40 |
GPT-6 Astra and Claude Fable 5.1 build their lead in Interactive Control and Perception and Estimation, the two workflows that depend on reading multimodal observations and acting on repeated feedback; the authors attribute the gap to visual grounding. Scores compress in Policy Development, which resembles conventional ML engineering. Mechanical Design humbles everyone: every submitted design scores zero on the worst-arm static stability margin checkpoint. Most designs pass structural validity, then fail under load and in-motion collision.
More numbers worth keeping. GPT-6 Astra scores 92.6 on bin clearing with 100% clearance at 9.22 parts per minute. On the RoboTwin robustness track the best final score is 20.0, and official scores land 19 points (LIBERO) and 41 points (RoboTwin) below development scores on perturbation subtasks; agents that build their own perturbation proxies still fail to close the gap. In 29 of 36 nanoVLA sessions the submitted artifact beat the agent's first development-time evaluation, so feedback does drive improvement, bounded by what the feedback can measure. Richer scaffolding raises scores and cuts costs for most models, but GPT-6 Astra is already strong at the bare L1 API and occasionally loses points when support is added. In the method-agnostic pose task, GPT-6 Astra was the only agent to actually use the GPU, training a small CNN as a fallback for unreliable depth.
This is the first benchmark that scores the robotics development stack end to end for coding agents, and per-workflow profiles expose where each model is lopsided. The economics also reframe model choice: DeepSeek-V4.1-Flash at $0.40 per task with an RLE Index of 38.9 sits in a different value class from a flagship at $39. For agent training, the 41-point robustness drop is the most valuable datapoint in the paper. Passing at development time does not mean passing under perturbation, and physically grounded hidden tests are precisely the signal current training environments do not provide.
The authors name three: simulation-only results say nothing about real-world reliability or safety; the four workflows cover only part of what robot learning engineers do; public release invites training-data contamination. Reading the appendix adds more. The RLE Index aggregates heterogeneous verifier rewards rather than a uniform success rate, so cross-family comparisons need care. Each task reports a single selected run with no variance. Grok 4.6 and Gemini 3.7 Flash lack token and cost data because their harnesses were incompatible with Harbor, which weakens confidence in their placement. And the three GPT-5.6 variants span 32 points, from 48.7 down to 16.3, a healthy reminder that single-run noise here is substantial.