3,600-run experiment shows AI agents favor big-cloud tools over technically optimal picks
Santiago Víquez ran a large-scale AEO (generative engine optimization) experiment: he gave four frontier models from the Claude and GPT families 10 real document-processing tasks (e.g., invoice extraction, redline contract reading) without naming any tool in the prompts, running 3,360 trials to see which API each model recommends by default. All prompts and data are public. The chosen track was document-processing APIs—a market where giants like Microsoft, Google, and AWS compete fiercely with startups such as Reducto and Extend.
Confirmed
- When directly asked to recommend a tool, 79% of recommendations went to big-cloud APIs (Azure Document Intelligence, Google Document AI, etc.)
- In coding scenarios, models picked big-cloud services on their own 70% of the time
- The conclusion: models don't always recommend the technically best tool, and the author argues that in the agent era, recommendations need human verification—verifiable capability claims will win in product marketing
Why it matters
- As agents increasingly make product choices on users' behalf, "recommendation bias" becomes a new industry issue: favoring big vendors may silence technically superior smaller products in the agent channel
- For product teams, agent-oriented marketing (AEO) will require verifiable evidence of capability rather than marketing copy
2026-11-17 ~ 2026-11-17 · 5 related posts
Primary sources
- I ran 3,360 experiments to see if AI agents recommend the right products — santiviquez · 2026-11-17
- [source] 3,360 experiments show LLMs steer 79% of API picks to big clouds, 70% when coding — santiviquez · 2026-11-17
- Why the experiment matters: do agents pick the best tools, or do we still need verification? — santiviquez · 2026-11-17
- [source] 3,360 experiments test which document-processing tool AI agents actually recommend — santiviquez · 2026-11-17
1 near-duplicate retellings: santiviquez