AutoResearchExam uses hidden test sets to study how AI agents do 24-hour research
AlexGDimakis · x · 2026-09-10
Alex Dimakis introduces AutoResearchExam: keep a hidden test set and observe how models perform during 24-hour autonomous research tasks. The proposed metric AUARC (Area Under the Auto Research Curve) measures the hidden test reward curve, capturing overfitting vs. careful research behavior.
Notable findings:
- GPT-5.6 Sol spent over half the observed research rounds tuning hyperparameters
- Fable and Opus tuned far less, instead pursuing new ideas or substantive fixes
- Fable and Opus tested several variants locally before committing
More from coding & agent
- IDA 9.5 ships Qualcomm Hexagon decompiler plus IDA MCP for AI-driven reverse engineering — OwariDa · 2026-10-27
- For OSS apps, proofs of concept — not PRs to one repo — may be the core primitive — dbreunig · 2026-10-27
- Hook everything into the Dot: a workaround to stretch ChatGPT subscription limits — teortaxesTex · 2026-10-27
- Jev launches people-search API for agents at $0.0089/lead, claiming 85% cheaper than Clay — alexcovo_eth · 2026-10-27
- Going through every compound engineering skill to decide where humans stay in the loop — kieranklaassen · 2026-10-27
- UCSB CS Prof. Xifeng Wang lands 3 grants worth $180K+ for practical agentic AI research — xwang_lk · 2026-10-27