3,360 experiments: agents pick the tools they know, not the best ones

santiviquez · x · 2026-11-17

Santiago Víquez ran 3,360 experiments to see whether AI agents recommend technically the best tools — probing how product marketing changes when agents become the buyers.

Setup: document processing APIs (Azure, Google, AWS, Reducto, Extend, LlamaParse, plus Anthropic, OpenAI, Mistral) across 10 use cases (financial tables, contracts, invoices, receipts, forms), 4-5 prompt variations each.

Key findings

Takeaway: when agents actually verify claims, the best product wins — but today's AEO tricks can still game recommendations. Open question: will models check benchmarks/docs by default?

Related event: 3,600-run experiment shows AI agents favor big-cloud tools over technically optimal picks(5 posts)→

Original post →

More from AGI Musings

AGI Musings channel →