Smaller models aren't always cheaper: the hidden costs LLM business cases miss
zeuslac · reddit · 2026-09-22
The author argues most AI business cases only track the model bill and miss where cheap-per-call models actually make workflows more expensive.
The mechanism:
- On easy cases a small model works and the saving is real. On hard cases its suggestions are weak enough that reviewers check them against source documents and redo the work — slower than no AI at all.
- Errors that slip through review cost far more to fix later than inference ever cost. The model bill drops while total work cost rises, and nobody notices because review time sits in a different system.
Two counterintuitive takeaways:
- Difficulty-based routing's value depends entirely on the price gap; a single price drop can eliminate the reason for the routing layer, so keep it cheap to unwind.
- A router can be financially better yet fail on quality: send enough hard cases to a weak model and the overall error rate can exceed the manual baseline even while the spreadsheet shows savings.
The fix is unglamorous: one record per task tying model usage, review time and later corrections together, so you see the full cost of a completed task rather than the cost of a call.
Related event: Small Models May Cost More: Hidden Rework and Review Costs(4 posts)→
More from AGI Musings
- Even With Powerful Proof Machines, Mathematicians' Value Stays Immense — cloneofsimo · 2026-09-22
- cloneofsimo: academia badly underestimates the problems OpenAI's math agents are solving — cloneofsimo · 2026-09-22
- GovAI paper argues frontier AI evals should be embedded inside labs, not just API tests — HaydnBelfield · 2026-09-22
- MIT Press makes its AI & ML book library 100% free — 9 classics to start with — thisguyknowsai · 2026-09-22
- DeepMind's Hassabis: AI could be 10x the industrial revolution — arriving in a decade, not a century — victor_explore · 2026-09-22
- Google's ScientistTwo solves 80.4% of 107 top-venue ML problems autonomously — thisdudelikesAI · 2026-09-22