Why the 15-minute time limit: bench author says it matches median agent runtime in prod

JoshPurtell · x · 2026-11-18

JoshPurtell explains his autoresearch benchmark design: it has a 15-minute time limit to correlate with real production usefulness, since the median agent runtime is around 15 minutes.

This is why low effort scored best with GPT-6.2 Astra — higher effort settings didn't get enough attempts within the limit. He adds that max effort would "almost certainly be disgusting" (i.e., potentially far stronger).

Related event: Insider Details Internal Autoresearch Benchmark with 15-Minute Limit(2 posts)→

Original post →

More from coding & agent

coding & agent channel →