I ran 3,360 experiments to see if AI agents recommend the right products
santiviquez · x · 2026-11-17
Santiago Víquez ran 3,360 experiment runs testing whether today's agents recommend the best product for a job. He picked the crowded document-processing API space—big clouds (Microsoft, Google, AWS), startups (Reducto, Extend, LlamaParse), and AI labs (Anthropic, OpenAI, Mistral)—and designed 10 use cases (financial tables, legal contracts, invoices, receipts, paper forms) with 4–5 prompt variations each: some asking directly for a recommendation, some starting from the problem, some asking the model to write implementation code instead.
His thesis: as agents become the new buyers, product marketing will change, and products with verifiable claims will win over those without.
More from coding & agent
- arXiv paper 'Agentic Computational Algebra' benchmarks agents on structure discovery and reuse — rohanpaul_ai · 2026-11-17
- UCLA and AWS paper: AI agents find math shortcuts but thinking time often cancels the gains — rohanpaul_ai · 2026-11-17
- Pixel art + Gaussian splats game experiments: all code, no assets, but roadblocks ahead — algo_diver · 2026-11-17
- Fable 5.2 praised as first model that ignores unimportant feedback and tells you to ship — neuroecology · 2026-11-17
- OpenAI's jxnlco on building taste: 'If you want good taste, you have to eat' — jxnlco · 2026-11-17
- Together AI and Ollama explain how prefix caching keeps long agent sessions fast — togethercompute · 2026-11-17