RExBench: best coding agent completes only 33% of autonomous research extensions
najoungkim · x · 2026-10-16
RExBench, an ACL 2026 paper by Sebastian Schuster, Najoung Kim and colleagues, introduces a benchmark testing whether coding agents can autonomously implement AI research extensions — validating novel research hypotheses on top of existing papers and codebases.
Design and findings:
- 12 realistic research-paper extension tasks, each built on an existing paper and codebase with domain expert-written instructions
- Robust to data contamination, with automatic evaluation infrastructure that executes agent outputs to check success criteria
- 12 LLM agents across two frameworks (aider and OpenHands) were evaluated: all fail to autonomously implement the majority of extensions, with the best agent at 33% success rate
- With human-written hints, performance improves but the best remains below 44%
Conclusion: current agents are still short of handling realistic research extension tasks without substantial human guidance.
More from coding & agent
- Running a multi-model UI design arena: Paseo orchestrates four design drafts per model — vista8 · 2026-10-16
- Cortex A/B-tests coding-agent skills and rules by replaying developer corrections before keeping them — Samuele Giampieri · 2026-10-16
- Tool calls are the hidden bottleneck killing AI coding speed — Crabbox and 5 fixes — steipete · 2026-10-16
- Agentic sandbox WorldBox doubles GPT-6 Astra's MazeBench score, 23 to 61 gems — ZhitingHu · 2026-10-16
- Two prompts: Gemini 4 Argon turns a single-player game into full multiplayer — chongdashu · 2026-10-16
- Anthropic and Gray Swan launch $300,000 indirect prompt injection red-teaming contest — bcherny · 2026-10-16