RExBench: best coding agent solves only 33% of autonomous research extension tasks
najoungkim · x · 2026-10-26
An ACL paper introduces RExBench, a benchmark of realistic extensions to 12 research papers testing whether coding agents can autonomously implement AI research extensions. Tasks build on existing papers and codebases with expert-written instructions, are robust to contamination, and use automatic execution-based evaluation. Across 12 agents built with aider and OpenHands, none solved the majority of extensions: the best reached 33% success, staying below 44% even with human-written hints—showing agents still fall far short of handling realistic research extension without substantial human guidance.
More from coding & agent
- Claude rebuilds a Gaussian-splatted garden scene in Blender with Geometry Nodes — CSProfKGD · 2026-10-26
- 50 agent runs, zero doc reads: five models pass tests but skip the documentation — ducha_aiki · 2026-10-26
- Every's 'frontier team' model: 100 AI experiments are useless if none land — every · 2026-10-26
- Fan-built AI site curates 250 best-under-$250 picks from 1,300 replies to Lenny's tweet — Scobleizer · 2026-10-26
- Nat Eliason shares deep-dive on how to set up your bot for maximum value — nateliason · 2026-10-26
- 6 critical production AI agent failures, demonstrated via an organ transplant agent — MaryamMiradi · 2026-10-26