Why the 15-minute time limit: bench author says it matches median agent runtime in prod
JoshPurtell · x · 2026-11-18
JoshPurtell explains his autoresearch benchmark design: it has a 15-minute time limit to correlate with real production usefulness, since the median agent runtime is around 15 minutes.
This is why low effort scored best with GPT-6.2 Astra — higher effort settings didn't get enough attempts within the limit. He adds that max effort would "almost certainly be disgusting" (i.e., potentially far stronger).
Related event: Insider Details Internal Autoresearch Benchmark with 15-Minute Limit(2 posts)→
More from coding & agent
- Microsoft's Autopilot agent Sebastian, born from a Christmas side project, demos real work at Ignite — donasarkar · 2026-11-18
- Why Cheaper Tokens Can Still Mean a Bigger AI Bill: The Agent Call-Stacking Problem — samcharrington · 2026-11-18
- DAWE edits 5,064 of 4B weights to fix harness mismatch, no retraining needed — lateinteraction · 2026-11-18
- Open-source vision-demos repo (1.2k stars): golf swing analysis with ViTPose+ and SAM 3.1 — spillai · 2026-11-18
- Agent claimed it edited my Obsidian vault but didn't — took three tries and a confrontation — HankYeomans · 2026-11-18
- Hypertune open-sources KV-cache soft-prefill adapters to serve heterogeneous fine-tunes in one batch — JoshPurtell · 2026-11-18