o3's time horizon hits ~110 minutes yet no system clears 31% on DeepScholar-Bench
le_james94 · x · 2026-09-16
Noting that DeepSeekMath-V2 makes verification the product, this post highlights how measurement is splitting in two: METR puts o3's 50% time horizon near 110 minutes of human work, doubling roughly every 7 months, while on DeepScholar-Bench no tested system clears a 31% geometric mean at writing related-work sections. Both numbers are real.
Related event: Three Weeks with Stanford CS329A: Generators Have Outrun Verifiers(9 posts)→
More from Research
- Detecting covert instruction-following via internal representations parallels J-space work — dioscuri · 2026-11-02
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Mistral Large 4 writes a quantum chemistry SCF solver from scratch with one short prompt — MistralAI · 2026-11-02
- Hunting for the most concrete "curing cancer with AI" roadmap of 2025 — iskander · 2026-11-02
- New theory paper proves linear decay is the unique minimax-optimal stepsize schedule for SGD — konstmish · 2026-11-02
- Karpathy's nanochat: the best ChatGPT $100 can buy, near 60k GitHub stars — karpathy · 2026-11-02