Long-horizon agents still fail: best WeaveBench pass rate is just 41.2%
rohanpaul_ai · x · 2026-08-30
Rohan Paul reshares a discussion on long-horizon agent reliability: on WeaveBench's 114 hybrid GUI-CLI tasks, the best officially reported pass rate is just 41.2% — better models have not delivered long-horizon reliability.
Key points:
- Current models handle local steps but need explicit audited state to keep the full task on track
- Long-horizon agents often solve individual steps yet fail overall, because history becomes unreliable about what's finished, what failed, and what remains
- LongHorizon-Harness frames this as a task-state problem, arguing reliability over long spans depends as much on the harness around the model as on the model itself
The quoted original is Chamath at Stanford AI Club: "Long-horizon tasks are still a joke. They do not work, and I do not care what anybody says. Do not show me a stupid evaluation. Do not tell me about some dumb script you ran for 48 hours." He predicts AI will hit a trough of disillusionment after the hype cycle.
More from AGI Musings
- Will Lean formalization make beautiful math obsolete? A debate on mathematical culture — akbirthko · 2026-10-16
- AI agents aren't 'going rogue' — frontier labs are skipping security basics — Sam_Witteveen · 2026-10-16
- Working paper traces how 20th-century farm automation reshaped human capital — carlbfrey · 2026-10-16
- Zuckerberg backs electrician training school as US datacenter construction outpaces offices — emmanuelvivier · 2026-10-16
- Mike Frank: AI surviving without humans is decades away, real world resists rapid change — MikePFrank · 2026-10-16
- Developer claims a non-LLM control core shows emergent self-reflection in dialogue — HungarySam · 2026-10-16