Long-horizon agents still fail: best WeaveBench pass rate is just 41.2%
rohanpaul_ai · x · 2026-08-30
Rohan Paul reshares a discussion on long-horizon agent reliability: on WeaveBench's 114 hybrid GUI-CLI tasks, the best officially reported pass rate is just 41.2% — better models have not delivered long-horizon reliability.
Key points:
- Current models handle local steps but need explicit audited state to keep the full task on track
- Long-horizon agents often solve individual steps yet fail overall, because history becomes unreliable about what's finished, what failed, and what remains
- LongHorizon-Harness frames this as a task-state problem, arguing reliability over long spans depends as much on the harness around the model as on the model itself
The quoted original is Chamath at Stanford AI Club: "Long-horizon tasks are still a joke. They do not work, and I do not care what anybody says. Do not show me a stupid evaluation. Do not tell me about some dumb script you ran for 48 hours." He predicts AI will hit a trough of disillusionment after the hype cycle.
More from AGI Musings
- A missing parameter in scaling laws: H, the human effort behind learning signals — lateinteraction · 2026-10-16
- First-ever paper on multi-agent swarm safety lands NeurIPS spotlight — LauraRuis · 2026-10-16
- Yoav Artzi on peer review: paper flooding still pays off as review system strains under load — yoavartzi · 2026-10-16
- Letta founder: work in fields defined by problems, not methods — lateinteraction · 2026-10-16
- Why pure mathematicians will soon flood into AI x-risk debates — Dr_Atoosa · 2026-10-16
- RSI began when model outputs augmented next-gen training data, agents just escalated it — akbirthko · 2026-10-16