OpenAI internal R&D tasks show 15-minute automation horizon vs METR's 3 hours, a 12x gap
srchvrs · x · 2026-10-24
A thread by srchvrs compares two estimates of AI agent "time horizon":
- METR benchmark: frontier models achieve 80% autonomous success on software tasks equivalent to 3 hours of human time.
- OpenAI internal R&D tasks: at the same 80% success threshold, the horizon drops to 15 minutes of human-equivalent time.
That's roughly a 12-fold difference. The author cautions this isn't a controlled comparison (different tasks, eval methods, possibly different models), but suggests (1) METR tasks may be too easy and unrealistic, and (2) real-world agent reliability is far lower than METR's estimates imply.
He clarifies the 15-minute figure refers to estimated human completion time, not agent runtime, and uses essentially the same interpretation as METR — so the comparison is apples-to-apples. The widely circulated claim that models can handle multi-hour or multi-day tasks may significantly overstate current agent automation capability.
Related event: OpenAI data shows AI agents reliably work only 15 minutes(2 posts)→
More from Models
- Kimi K3.1 Pro scores just 9% on real-robot tasks, trailing Opus 5.5 and Gemini 4 Argon — peterwildeford · 2026-10-24
- Researchers call out 'hacking Claude' stories: asking basic cyber questions gets flagged — max_paperclips · 2026-10-24
- Claude consistently embodies itself as the Sun in AI-generated music videos — jam3scampbell · 2026-10-24
- Kimi K3.1 Pro cost breakdown: high mode $0.55/5min, max mode $0.95/11min — cedric_chee · 2026-10-24
- Hands-on with Kimi K3.1 Pro: 2-3x faster, $0.55 per task, closing on frontier 3D work — cedric_chee · 2026-10-24
- Kimi K3.1 Pro tested: 2-3x faster at $0.55 per task, open models close on frontier 3D — cedric_chee · 2026-10-24