I ran 3,360 experiments to see if AI agents recommend the right products

santiviquez · x · 2026-11-17

Santiago Víquez ran 3,360 experiment runs testing whether today's agents recommend the best product for a job. He picked the crowded document-processing API space—big clouds (Microsoft, Google, AWS), startups (Reducto, Extend, LlamaParse), and AI labs (Anthropic, OpenAI, Mistral)—and designed 10 use cases (financial tables, legal contracts, invoices, receipts, paper forms) with 4–5 prompt variations each: some asking directly for a recommendation, some starting from the problem, some asking the model to write implementation code instead.

His thesis: as agents become the new buyers, product marketing will change, and products with verifiable claims will win over those without.

Original post →

More from coding & agent

coding & agent channel →