3,360 experiments test which document-processing tool AI agents actually recommend

santiviquez · x · 2026-11-17

An AEO (answer-engine optimization) study: the author asked four frontier Claude and GPT models to handle 10 real document-processing jobs (e.g., invoice extraction, redlined contracts) without naming any tool — 3,360 experiments total — recording every prompt, response and result.

The evaluation framework is open-sourced on GitHub under MIT with full code, tasks and results. The open question: will models eventually verify against benchmarks by default, or will today's AEO tricks still game recommendations?

Related event: 3,600-run experiment shows AI agents favor big-cloud tools over technically optimal picks(5 posts)→

Original post →

More from Research

Research channel →