UT Austin releases AutoResearchExam, a 24-hour open-ended benchmark for AI research agents
simonguozirui · x · 2026-09-10
The UT Austin team (Alex Dimakis et al.) released AutoResearchExam, a benchmark for open-ended ML and engineering tasks: it spans seven research areas including model training, data curation, AI safety and interpretability. Each task gives an agent a CPU/GPU machine for 24 hours to iterate through experiments, scored on combined speed and quality. Its unique test: whether agent improvements hold up on unseen data — and they find research agents often overfit. Benchmarking is supported via Harbor, with a head-to-head comparison at the frontier including Astra.
More from coding & agent
- Cursor launches Agent SDK: agents as code with Markdown prompts and TypeScript tools — soleio · 2026-10-27
- Vibecheck, the 'usable formal methods' hackathon, sells out two floors in SF — spikedoanz · 2026-10-27
- How do you verify AI agent outputs beyond unit tests and LLM-as-judge? — Practical_Green5775 · 2026-10-27
- AI agents may soon become the only way to run code, not just write it — IgorCarron · 2026-10-27
- Hermes Agent used as browser QA tester finds 8 issues in a bug-seeded site — techlatest_net · 2026-10-27
- Open-source FableCut lets AI agents edit a real video timeline via remote MCP — FastPresence9799 · 2026-10-27