220-question test: 70% of model web searches were unnecessary, cost 75x more tokens
demian_ai · x · 2026-10-15
A $20 experiment ran 220 hand-verified questions across 11 categories with web search on and off, yielding 1,760 answers from gpt-5.4-mini.
Key findings
- The mini model searched on 39% of questions, but 70% of those searches were for answers it already got right without searching — including looking up the GDPR start date it already knew
- Over-searching varies sharply by category: law questions triggered searches 78% of the time (97% unnecessary); every single search in medicine and plant ID was unneeded; math, stable code, and general knowledge never triggered a search
- A search never turned a correct answer wrong (0 out of 880), but it doesn't fix stale sources either
The cost angle: with no tool enabled a request costs 58 input tokens; with the search tool on but unused it's 4,500. The takeaway: auto search triggering is far from precise and quietly burns money in per-token billing. The attached chart breaks down accuracy shifts, search rates, and unneeded-search percentages per category.
More from coding & agent
- Datadog's Scout 1.0: post-trained GLM-5.3 Flash matches frontier models for AI SRE at lower cost — Tim_Dettmers · 2026-10-16
- Anthropic ditched Git as agents pushed merges to 30,000 a day, up 10x in 4 months — shashib · 2026-10-16
- LlamaIndex launches ExtractBench Pro to tackle the long tail of document extraction — llama_index · 2026-10-16
- Stop defaulting to Medium: benchmarking Gemini 4 Argon vs 3.8 Flash effort budget token costs — rseroter · 2026-10-16
- Critical 9.5/10 RCE hits Next.js use cache; patch 16.4.1 now — jedisct1 · 2026-10-16
- Anthropic now merges 30,000 code changes a day, reviews code by 'intent' not lines — Miles_Brundage · 2026-10-16