AutoResearchExam: a 24-hour benchmark finds AI research agents overfit their improvements
AlexGDimakis · x · 2026-11-10
UC Berkeley's Alex Dimakis team released AutoResearchExam (ARE), a benchmark for open-ended ML and engineering tasks. v1.1 now runs natively in the Harbor framework with any agent and any model via a single harbor run -d bespokelabs/autoresearch-exam command.
- Covers seven research areas including model training, data curation, AI safety and interpretability
- Each task gives an agent a CPU/GPU machine for 24 hours to iterate through experiments and feedback, scored on a combined speed + quality metric
- Unique design: checks whether agent improvements hold up on unseen data — finding that AI research agents frequently overfit while optimizing
More from coding & agent
- Ponytail, the Claude Code Plugin With 188k Stars, Makes Agents Delete Before They Write — nevrekaraishwa2 · 2026-11-11
- Ponytail adds repo-level audit that ranks issues instead of just listing them — nevrekaraishwa2 · 2026-11-11
- 53% Less Code: Ponytail Forces Agents to Ask 'Does This Need to Exist?' Before Writing — nevrekaraishwa2 · 2026-11-11
- Ponytail 5 cuts Claude Code output 53% and cost 26% across 5,700+ real sessions — nevrekaraishwa2 · 2026-11-11
- YC F26 Startup Elimentary Launches AI Agents for Private Credit Ops, Cuts NAV Checks to 30 Minutes — ycombinator · 2026-11-11
- Carbon test finale: tactical FPS with bot AI that tests and repairs its own work — kimmonismus · 2026-11-11