How the experiment was built: 10 document-processing jobs, 4-5 prompt variants each
santiviquez · x · 2026-11-17
The author describes the experiment design: 10 use cases covering financial tables, legal contracts, invoices, receipts and paper forms, each with 4-5 phrasing variants — direct recommendation requests, problem-first framings, casual asks — plus variants asking models to write the implementation code, to test whether phrasing changes what gets recommended.
Related event: 3,600-Query Experiment: 79% of API Recommendations Go to Big Cloud(2 posts)→
More from Research
- 8th NeurIPS Robot Learning Workshop to debate: Is Physical AI truly going zero-shot? — m_wulfmeier · 2026-11-17
- François Fleuret dissects a fundamental transformer limitation: sequence and token indexing — francoisfleuret · 2026-11-17
- François Fleuret explains a memory trick: pay O(time steps) instead of O(steps × components) — francoisfleuret · 2026-11-17
- Fudan & OpenMOSS map the trade-off: long-context hybrids learn more, extrapolate less reliably — jiqizhixin · 2026-11-17
- Models favor their own APIs in code: Opus calls Claude API in 97% of implementations — santiviquez · 2026-11-17
- Search behavior gap: Opus searched in 10/60 answers, GPT models searched every time — santiviquez · 2026-11-17