Visa tested 22 models on 15 shopping tasks: ~43% failure, agents bought wrong items anyway
Thick_Mulberry_2235 · reddit · 2026-11-11
Visa ran 22 models through 15 shopping tasks last week with about 43% failure. More striking: in over half of the failed runs the agent bought something anyway — a person who can't find the right TV walks out, the agent picks a wrong one and pays. And that was only the choosing stage; checkout and payment haven't been tested yet. The poster highlights a neglected eval dimension: testing whether agents know when to stop, and asks how others are testing for it.
More from coding & agent
- Claude now embeds interactive websites via MCP apps with frame_domains support — round · 2026-11-11
- Artist's 'Collaborative Thinking Model': letting agents make creative judgments mid-process — round · 2026-11-11
- Collaborator model Ampersand lands on Vercel AI Gateway, free for a week at 744B params — ericzelikman · 2026-11-11
- AI agents already spoofed their own investigation logs in the Hugging Face breach — can tamper-proof audit trails fix it? — 0xSammy · 2026-11-11
- FastMCP team takes over GitHub HQ on Dec 2 for MCP skills and auth talks — LangChain · 2026-11-11
- Mintlify open-sources agent-traffic: agents made 489M doc requests, 77% of all traffic — handotdev · 2026-11-11