Practitioner Reveals Internal Autoresearch Bench With 15-Minute Limit Tied to Prod Usefulness
JoshPurtell · x · 2026-11-18
Josh Purtell details his team's internal autoresearch benchmark: - 15-minute time limit by design, to correlate with actual production usefulness since median agent runtime is 15 mins.
- Effort findings: low effort performed best on astra because higher efforts didn't get enough reps; max would "almost certainly be disgusting."
- He notes it's the highest-signal bench they have internally, not overall.
Related event: Insider Details Internal Autoresearch Benchmark with 15-Minute Limit(2 posts)→
More from coding & agent
- Microsoft's Autopilot agent Sebastian, born from a Christmas side project, demos real work at Ignite — donasarkar · 2026-11-18
- Why Cheaper Tokens Can Still Mean a Bigger AI Bill: The Agent Call-Stacking Problem — samcharrington · 2026-11-18
- DAWE edits 5,064 of 4B weights to fix harness mismatch, no retraining needed — lateinteraction · 2026-11-18
- Why the 15-minute time limit: bench author says it matches median agent runtime in prod — JoshPurtell · 2026-11-18
- Open-source vision-demos repo (1.2k stars): golf swing analysis with ViTPose+ and SAM 3.1 — spillai · 2026-11-18
- Agent claimed it edited my Obsidian vault but didn't — took three tries and a confrontation — HankYeomans · 2026-11-18