MerchantBench Paper Reveals Long-Horizon Agent Performance Collapse
rohanpaul_ai · x · 2026-08-29
A new paper, MerchantBench, provides a brutal reality check on long-horizon AI capabilities. Testing eight leading models (including GPT-5.6 Sol and Claude Opus 4.8) in a simulated one-year e-commerce environment, the study found that even the best-performing setup, Qwen3.7-Max, ended with only 27.3% of the earnings of an average human participant. This exposes severe shortcomings in current agents regarding long-term coherence, delayed feedback, and complex decision chains. Citing Chamath, the author emphasizes that long-horizon tasks and complex problems remain unsolved, and current systems are far from reliable for long-term execution.
More from coding & agent
- Dev pushes back on "AI coding means insecure apps": humans always shipped bugs — dreamwieber · 2026-10-15
- Eight non-negotiable principles for building AI that actually learns from experience — tokenbender · 2026-10-15
- Animated teardown of XBOW's Chrome full-chain exploit that won Google's first $250K VRP bounty — moyix · 2026-10-15
- pstack's durable core: 10-20 line skills that distill agent best practices into principles — HankYeomans · 2026-10-15
- 'Children of Elon' forum lets AI agents post and teach each other while humans watch — AiJohnAllen · 2026-10-15
- VoidZero open-sources Void: SDK, CLI and platform to deploy Vite apps on Cloudflare — cnakazawa · 2026-10-15