MerchantBench: 365-Day E-Commerce Simulation Tests LLM Agents' Long-Term Coherence
rohanpaul_ai · x · 2026-10-02
A team unveils MerchantBench, a 365-day order-level simulation for seller-side e-commerce built on 98,843 real product records with 26 interaction tools, benchmarking LLM agents' Long-Term Coherence — sustaining purposeful behavior over extended horizons while adapting to accumulated evidence. It couples promptly observable upstream supplier events with delayed downstream order outcomes, requiring agents to track order lifecycles and revisit earlier decisions across sourcing, listing/pricing, and cash-flow management. Eight LLMs were evaluated under two agent frameworks in 48 runs of 365 simulated days, with models generally struggling at long horizons.
More from coding & agent
- Claude Code overtakes Cursor and Copilot in job-post mentions ahead of Anthropic IPO — antgoldbloom · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- Orchestrator's Cloud Agents graduate from beta to general availability — julianweisser · 2026-11-19
- Zero users, 33 pages shipped: dev crowdsources audit for AI-agent trust schema — TheHamzaShahbaz · 2026-11-19
- She vibecoded her own photobooth with Claude Code — and shared the prompt — toolstelegraph · 2026-11-19
- Should AI Agents Write to Your Memory or Only Read? Dev Weighs In — jordimr · 2026-11-19