Routing simple requests to a cheap model made our total LLM bill worse
Massive_Tell_4276 · reddit · 2026-09-17
A developer shares a counterintuitive cost lesson: routing simple requests to a cheaper model tier lowered average cost per call, but total spend rose.
The cheap model often returns low-confidence outputs, so the system re-runs retrieval with the same context and falls back to the expensive model. With a loose confidence threshold, this cascade fires often, and context duplication means one resolved task pays for two prompts. The dashboard was accurate per call yet led to the wrong overall conclusion.
The author is rebuilding analysis around cost per resolved task, fallback rate, and latency across the full cascade, plus cohort slices since a few request types cause most double runs. He asks how others tune confidence thresholds without hurting completion rates, and whether retrieval cost should be counted once or per stage.
More from coding & agent
- Typesafe's jev ships a live, playable demo for developers — holdenmatt · 2026-09-17
- TypeSafe AI cookbook: line-by-line semantic search scoring 218 lines in one request — hackgoofer · 2026-09-17
- 'Vibe automating': offloading admin work — lead scoring, reporting, email workflows — to AI — aronkor · 2026-09-17
- Why AI coding assistants keep generating deprecated functions, and how devs cope — sharificles · 2026-09-17
- Bend2 launches tomorrow: Victor Taelin's 'post-AGI' language ditches interaction nets for speed — Birchlabs · 2026-09-17
- Devin's Fable 5.1 + SWE-2 Fusion Mode Runs 3+ Hours on Just 1% of Usage, Early User Reports — CtrlAltDwayne · 2026-09-17