AutoResearchExam benchmark: AI research agents overfit in 24-hour open-ended tasks
dair_ai · x · 2026-09-10
The team behind AutoResearchExam released a benchmark testing agents on open-ended ML and engineering research: 7 areas including model training, data curation, AI safety and interpretability.
- Each task gives an agent 24 hours with a CPU/GPU machine; speed and quality are combined into one score
- Unique twist: improvements must hold up on unseen data — and agents frequently overfit while trying to improve
- Frontier head-to-head: Astra starts strong and leads for up to 19 hours before Fable 5.1 catches up; Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 sit on the cost-performance Pareto frontier
- Key finding: agents stagnate from low-quality exploration — possibly OOD effects or a lack of the intuition-driven creativity top human researchers have
More from Research
- Physicist lets Claude Opus pick its own research question — result validated by experts — IgorCarron · 2026-10-27
- FlexGlue: sparse-to-dense matching that beats dense accuracy at a fraction of the cost — zhenjun_zhao · 2026-10-27
- Mundo turns a few room photos into an explorable, simulatable 3D world — at 1.5 FPS — mark_k · 2026-10-27
- Mathematics as an experimental science: Marc Lelarge to talk LLM-driven math experiments at ENS — marc_lelarge · 2026-10-27
- DOE Genesis Mission Phase II: AI for particle accelerators and rare isotope separation — prescott · 2026-10-27
- Tenstorrent runs Proteina protein model at high throughput, claims no HBM shortage — gajesh · 2026-10-27