AutoResearchExam: a 24-hour benchmark reveals how AI agents really do research
AlexGDimakis · x · 2026-09-10
Researchers release AutoResearchExam, a benchmark of open-ended ML and engineering tasks across seven areas (model training, data curation, AI safety, interpretability). Agents get a CPU/GPU machine and 24 hours per task to iterate via experiments. Key findings: GPT-5.6 Sol spent over half its observed rounds tuning hyperparameters, while Fable and Opus pursued new ideas or substantive fixes and tested variants locally before submitting. The team also studied behavior under hints and harness changes, and will maintain the benchmark toward RSI research.
More from coding & agent
- IDA 9.5 ships Qualcomm Hexagon decompiler plus IDA MCP for AI-driven reverse engineering — OwariDa · 2026-10-27
- Redditor builds a Star Trek-style home AI: the model was easy, the audio plumbing wasn't — FF2183 · 2026-10-27
- For OSS apps, proofs of concept — not PRs to one repo — may be the core primitive — dbreunig · 2026-10-27
- Hook everything into the Dot: a workaround to stretch ChatGPT subscription limits — teortaxesTex · 2026-10-27
- Jev launches people-search API for agents at $0.0089/lead, claiming 85% cheaper than Clay — alexcovo_eth · 2026-10-27
- Going through every compound engineering skill to decide where humans stay in the loop — kieranklaassen · 2026-10-27