AutoResearchExam: a 24-hour benchmark finds AI research agents overfit, with Fable 5.1 edging Astra
AlexGDimakis · x · 2026-09-10
Alex Dimakis's team releases AutoResearchExam, a benchmark of open-ended ML and engineering tasks spanning seven research areas including training, data curation, safety and interpretability.
- Each task gives agents 24 hours on a CPU/GPU machine to improve solutions via experiments; scoring combines speed and quality
- Unique twist: tests whether improvements hold on unseen data — agents often overfit while trying to improve
- Frontier race: Astra led for up to 19 hours before Fable 5.1 overtook for the top spot; Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 sit on the cost-performance tier
More from coding & agent
- Resin lets an agent build standalone LLM inference engines; vLLM and SGLang optimize DeepSeek kernels differently — AccBalanced · 2026-10-27
- Alchemy adds effect/Workflow support across Durable Objects, Workflows and Lambda Durable Functions — samgoodwin89 · 2026-10-27
- Alchemy's Lambda Web Functions run plain Node web servers on Lambda, no OpenNext needed — samgoodwin89 · 2026-10-27
- Alchemy Domain brings domain-driven design and story testing to Effect backends — samgoodwin89 · 2026-10-27
- Alchemy makes all ~2,850 AWS bindings work from any cloud, not just S3/Lambda/DynamoDB — samgoodwin89 · 2026-10-27
- Production AI agent scored 95% robustness at baseline, 37% under adversarial attack — njyx · 2026-10-27