Bespoke Labs launches AutoResearchExam: 29 tasks to benchmark 24-hour autonomous ML research agents
AashaySachdeva · x · 2026-09-10
- Bespoke Labs releases AutoResearchExam, a benchmark of 29 open-ended ML research tasks measuring how fast agents improve a private test score over a 24-hour sustained research window.
- Motivation: real research relies on iteration and exploration; one-shot and standard long-horizon evals miss this and can conflate validation-score gains with true generalization.
- Design: agents optimize a visible validation score while a hidden-test AUARC measures how quickly they find generalizing solutions.
- Findings: rankings shift with the evaluation window, showing a single time budget misses model differences; the report also analyzes cost efficiency and how agents allocate research effort.
- Runs on the standard Terminus 2 harness; tasks and leaderboard are public. Framed as an ingredient toward recursive self-improvement.
More from coding & agent
- Dev turns DJI Mavic into local AI security drone built with Opus 5.5 — alexcovo_eth · 2026-10-27
- Microsoft Research: ForkPilot cuts long-horizon agent compute by up to 59% — EliasEskin · 2026-10-27
- Cursor launches Agent SDK: agents as code with Markdown prompts and TypeScript tools — soleio · 2026-10-27
- Vibecheck, the 'usable formal methods' hackathon, sells out two floors in SF — spikedoanz · 2026-10-27
- How do you verify AI agent outputs beyond unit tests and LLM-as-judge? — Practical_Green5775 · 2026-10-27
- AI agents may soon become the only way to run code, not just write it — IgorCarron · 2026-10-27