AutoResearchExam: Testing AI Research Agents with 24-Hour Real-Machine Tasks

AutoResearchExam is a new benchmark for evaluating autonomous research ability in AI agents: each task gives an agent a CPU/GPU machine and a 24-hour window to work on open-ended machine learning and engineering problems. With 29 tasks across 7 research areas including model training, data curation, AI safety and interpretability, it offers a reproducible framework for measuring real "AI researcher" capability, and finds that agents commonly overfit.

Confirmed

• Setup: 29 open-ended tasks across 7 research areas; each task provides a CPU/GPU machine and a 24-hour window, with agents improving through experiments and feedback.
• Evaluation: a hidden test set is held out to observe 24-hour autonomous research; the AUARC (area under the Auto Research Curve) metric measures how fast an agent iteratively improves hidden-test-set scores rather than one-shot performance.
• Observations: per Alex Dimakis (m2), agent behaviors differ markedly across tasks; per dairai, agents commonly overfit.

Unconfirmed

• Attribution differs across posts: Alex Dimakis calls it his team's work, dairai and simonguozirui credit UT Austin (Dimakis et al.), while AashaySachdeva says Bespoke Labs released it. The paper should be treated as authoritative.

Why it matters

• The benchmark shifts evaluation from static Q&A to a real research setting—give a machine, give time, watch output—directly testing long-horizon, multi-iteration research ability.
• The hidden test set design reduces gaming and overfitting to public benchmarks, and AUARC provides a unified quantitative standard for measuring sustained improvement speed.

2026-09-10 ~ 2026-09-10 · 7 related posts

• [source] AutoResearchExam: a 24-hour benchmark finds AI research agents overfit, with Fable 5.1 edging Astra — AlexGDimakis · 2026-09-10
• [source] AutoResearchExam: a 24-hour benchmark reveals how AI agents really do research — AlexGDimakis · 2026-09-10
• Bespoke Labs launches AutoResearchExam: 29 tasks to benchmark 24-hour autonomous ML research agents — AashaySachdeva · 2026-09-10
• [source] AutoResearchExam uses hidden test sets to study how AI agents do 24-hour research — AlexGDimakis · 2026-09-10
• UT Austin releases AutoResearchExam, a 24-hour open-ended benchmark for AI research agents — simonguozirui · 2026-09-10

2 near-duplicate retellings: dair_ai · sudoraohacker