3,360 experiments: agents pick the tools they know, not the best ones
santiviquez · x · 2026-11-17
Santiago Víquez ran 3,360 experiments to see whether AI agents recommend technically the best tools — probing how product marketing changes when agents become the buyers.
Setup: document processing APIs (Azure, Google, AWS, Reducto, Extend, LlamaParse, plus Anthropic, OpenAI, Mistral) across 10 use cases (financial tables, contracts, invoices, receipts, forms), 4-5 prompt variations each.
Key findings
- Asked directly, models favor the most familiar vendors, not the strongest ones
- Self-recommendation bias is stark when writing code: Opus 5.5 called the Claude API in 58 of 60 implementations (97%) after recommending Claude in only 10 of 150 direct answers; GPT-6 Astra behaved similarly
- Asking the agent to verify against docs reshuffles the ranking: Extend, Content Understanding, Reducto and LlamaParse (covering 8-10/10 use cases per docs) rise to the top; heavily recommended Document Intelligence and Document AI fall out of the top 5
- Independent Openbenchmarks data (redlined contracts) confirms the reversal: Extend first at 93.1% accuracy, Reducto 84.8%, LlamaParse 83.8%, while top-recommended cloud products lag on real capability
Takeaway: when agents actually verify claims, the best product wins — but today's AEO tricks can still game recommendations. Open question: will models check benchmarks/docs by default?
More from AGI Musings
- Jarovsky: Those profiting from dangerous AI will never admit it's unsafe — LuizaJarovsky · 2026-11-17
- Gary Marcus mocks the ASI race: 'we're already in a mountain of shit' — GaryMarcus · 2026-11-17
- Ethan Mollick on the AI bull case: the expected value is just continued 2% growth — emollick · 2026-11-17
- Why the experiment matters: do agents pick the best tools, or do we still need verification? — santiviquez · 2026-11-17
- "Are all Reddit posts just AI now?" — users say everything reads like Claude — adonis_singh · 2026-11-17
- Beach birdwatching sparks an AI analogy: small focused models beat do-everything LLMs — shashib · 2026-11-17