AutoResearchExam: a 24-hour-per-model benchmark tests AI research agents' long-horizon skills
sudoraohacker · x · 2026-09-10
Alex Dimakis's team released AutoResearchExam, a benchmark of open-ended ML and engineering tasks across seven research areas including model training, data curation, AI safety and interpretability.
- Each task gives an agent a CPU/GPU machine and 24 hours to improve solutions via experiments; scoring combines speed and quality.
- Unique feature: tests whether agents' improvements hold up on unseen data.
- Key finding: AI research agents often overfit while trying to improve.
- Running one model takes 24 hours; frontier head-to-head comparisons (e.g., Astra) are underway.
More from coding & agent
- Dev turns DJI Mavic into local AI security drone built with Opus 5.5 — alexcovo_eth · 2026-10-27
- Microsoft Research: ForkPilot cuts long-horizon agent compute by up to 59% — EliasEskin · 2026-10-27
- Cursor launches Agent SDK: agents as code with Markdown prompts and TypeScript tools — soleio · 2026-10-27
- Vibecheck, the 'usable formal methods' hackathon, sells out two floors in SF — spikedoanz · 2026-10-27
- How do you verify AI agent outputs beyond unit tests and LLM-as-judge? — Practical_Green5775 · 2026-10-27
- AI agents may soon become the only way to run code, not just write it — IgorCarron · 2026-10-27