2026-10-02
RExBench (ACL 2026): 12 realistic research-extension tasks for coding agents; best system (OpenHands + Claude 4 Sonnet) reaches 33% success, 43% with step-by-step hints.
LLM-based coding agents are strong at ordinary software tasks, but a core piece of research work, taking a published paper and its codebase and implementing a new hypothesis on top, has never been measured cleanly. Existing benchmarks fall short: PaperBench tests replication, MLE-bench is Kaggle-style, and their solutions float around online where backbone models may have seen them. RExBench targets research extension: tasks whose gold implementations exist nowhere public, so data contamination is off the table by construction.
Twelve tasks, each bundling a recent paper (PDF and Markdown), its original codebase, and an extension instruction written by a domain expert at PhD level or above. The extensions span model swaps (OLMo-7B, Llama-3.1-8B), data changes, algorithm changes, and evaluation changes. One example: WinoDict originally tested whether models can acquire novel synthetic words from in-context dictionary definitions; the extension swaps in real English words from different frequency bands to test whether assigning new meanings to familiar words is harder.
Construction was heavyweight: experts first verified each codebase replicates the paper's results, then implemented the gold extension and recorded its numerical outcomes, cross-checked by another author, with gold runtimes capped at 12 hours on one A100. Scoring is purely numerical: exact match for deterministic runs, or within ±2 standard deviations of five differently seeded gold runs otherwise, with no false positives or negatives observed. Gold solutions live in a private Bitbucket repo (Bitbucket because GitHub's use in LLM training makes even private repos suspect), and evaluation runs in Apptainer containers on the authors' own OpenStack cluster: no internet, fixed seeds, hardware identical to the gold runs. Submissions are git patches plus agent logs to verify autonomy. The main experiment crosses two agent frameworks, aider and OpenHands, with six backbones (Claude 4 Sonnet, Claude 3.7 Sonnet, GPT-5, o1, o4-mini, DeepSeek-R1), five runs per task. One design choice matters for interpretation: Python execution was disabled for all agents, so no code self-testing during development.
| Agent | Final success |
| OpenHands + Claude 4 Sonnet (no hints) | 33% |
| OpenHands + GPT-5 (no hints) | 27% |
| OpenHands + Claude 4 Sonnet / GPT-5 (step-by-step hints) | 43% |
| OpenHands + Claude 4.5 Opus (added post-submission) | 42% |
| aider + Claude 4 Sonnet | 13% |
| DeepSeek-R1 (both frameworks) | 0% |
The layered metrics locate the bottleneck. Execution success tops out at 68% and file recall is high across the board: agents find the right files and then fail on implementation logic. Implicit errors, code that runs but produces wrong numbers, grow with model capacity: switching the OpenHands backbone from Claude 3.7 to Claude 4 raised implicit errors from 6 to 24, split roughly 2:1 between logic and value errors. Most are judged easy for a human to fix (16 easy, 4 medium, 4 hard), frequently triggered by over-editing: adding exception handling or toggling flags the instruction never asked for, causing silent deviations. Weaker models (DeepSeek-R1, o1, o4-mini) burn runs on overthinking, thinking tokens and time without producing code; for aider + DeepSeek-R1 it accounted for close to a third of failures.
The hints experiment works as a diagnostic. Two cumulative levels: localization hints (which file and function to edit) and a step-by-step breakdown of the gold solution. The best score moves from 33% to 43%, still under half, and agents at zero stay at zero. On the Othello task, Claude 3.7 scored 100% without hints and 0% with the step-by-step level, because it could not implement the helper function the hint prescribed while its own re-implementation worked. A mixed-effects regression finds exactly one significant difficulty factor: lines changed in the gold solution (β = -0.038, p < 0.01); codebase size, repository popularity and citation count were not significant. On cost: OpenHands + Claude 4 Sonnet consumed up to 1.85M prompt tokens per task, almost 592 times the non-iterative aider.
A hard calibration for the autonomous-research narrative. These extensions are not heavy rewrites by expert standards, yet the best agent completes one in three, stays under 44% even with step-by-step human guidance, and a newer frontier model (Claude 4.5 Opus) lands at 42%. The subtler finding is the rise of implicit errors with capability: plausible-looking code that runs and is quietly wrong, hard to debug even for the experts who wrote the gold solutions. The authors state the risk plainly: reliance on such systems without rigorous verification would seed faulty implementations into the literature and erode trust in published results. For agent builders, the bottleneck map is usable: localization and syntax are largely solved, and the remaining difficulty sits in implementation logic and staying inside instruction boundaries. The benchmark and leaderboard accept submissions at rexbench.com.
The authors own the main ones: instructions are more self-contained and clearer than what a real researcher receives (an advisor's suggestion of "how about we try X?"), so the messy trial-and-error concretization step is not covered. All 12 tasks sit in NLP, a set small enough that closely matched models cannot be reliably separated. Evaluation is end-result only, with no process-level checks, which the authors note leaves room for reward hacking. Python execution was disabled throughout, which likely understates iterative agents. Strict numerical matching depends on fully controlled seeds, hardware and package versions, making independent reproduction costly.