AutoResearchExam: a 24-hour benchmark finds AI research agents overfit, with Fable 5.1 edging Astra
AlexGDimakis · x · 2026-09-10
Alex Dimakis's team releases AutoResearchExam, a benchmark of open-ended ML and engineering tasks spanning seven research areas including training, data curation, safety and interpretability.
- Each task gives agents 24 hours on a CPU/GPU machine to improve solutions via experiments; scoring combines speed and quality
- Unique twist: tests whether improvements hold on unseen data — agents often overfit while trying to improve
- Frontier race: Astra led for up to 19 hours before Fable 5.1 overtook for the top spot; Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 sit on the cost-performance tier
More from coding & agent
- Redditor seeks real-world stacks for a persistent-memory personal AI that answers texts and emails — BrooksomidusGG · 2026-10-27
- Practical tip: use max effort for planning, xhigh for implementation in Claude Code — majidmanzarpour · 2026-10-27
- Chrome team explains WebMCP: agents get tools from your website, not a replacement for MCP — rseroter · 2026-10-27
- Agent-Native apps: why agents bypassing your UI can break business rules — rseroter · 2026-10-27
- 'My intelligence stack': developer splits AI coding work across Opus 5.5, Fable and Astra with human review gates — sh_reya · 2026-10-27
- Baseten webinar: evaluate open models for coding agents by benchmarking on your own merged PRs — baseten · 2026-10-27