RExBench: best coding agent solves only 33% of AI research extension tasks
najoungkim · x · 2026-10-16
RExBench (ACL 2026, Najoung Kim et al.) benchmarks whether coding agents can autonomously implement AI research extensions:
- Setup: 12 realistic extension tasks built on existing research papers and codebases, with domain expert-written instructions, contamination-robust, and an automatic evaluation infrastructure that executes agent outputs.
- Results: 12 LLM agents across two frameworks (aider, OpenHands) all fail to complete the majority of extensions — the best agent reaches 33% success.
- With hints: human-written hints help, but the best performance stays below 44%.
- Takeaway: current agents still need substantial human guidance for realistic research extension work.
More from coding & agent
- Running a multi-model UI design arena: Paseo orchestrates four design drafts per model — vista8 · 2026-10-16
- Cortex A/B-tests coding-agent skills and rules by replaying developer corrections before keeping them — Samuele Giampieri · 2026-10-16
- Tool calls are the hidden bottleneck killing AI coding speed — Crabbox and 5 fixes — steipete · 2026-10-16
- Agentic sandbox WorldBox doubles GPT-6 Astra's MazeBench score, 23 to 61 gems — ZhitingHu · 2026-10-16
- Two prompts: Gemini 4 Argon turns a single-player game into full multiplayer — chongdashu · 2026-10-16
- Anthropic and Gray Swan launch $300,000 indirect prompt injection red-teaming contest — bcherny · 2026-10-16