3,360 experiments test which document-processing tool AI agents actually recommend
santiviquez · x · 2026-11-17
An AEO (answer-engine optimization) study: the author asked four frontier Claude and GPT models to handle 10 real document-processing jobs (e.g., invoice extraction, redlined contracts) without naming any tool — 3,360 experiments total — recording every prompt, response and result.
The evaluation framework is open-sourced on GitHub under MIT with full code, tasks and results. The open question: will models eventually verify against benchmarks by default, or will today's AEO tricks still game recommendations?
More from Research
- 8th NeurIPS Robot Learning Workshop to debate: Is Physical AI truly going zero-shot? — m_wulfmeier · 2026-11-17
- François Fleuret dissects a fundamental transformer limitation: sequence and token indexing — francoisfleuret · 2026-11-17
- François Fleuret explains a memory trick: pay O(time steps) instead of O(steps × components) — francoisfleuret · 2026-11-17
- Fudan & OpenMOSS map the trade-off: long-context hybrids learn more, extrapolate less reliably — jiqizhixin · 2026-11-17
- Models favor their own APIs in code: Opus calls Claude API in 97% of implementations — santiviquez · 2026-11-17
- Search behavior gap: Opus searched in 10/60 answers, GPT models searched every time — santiviquez · 2026-11-17