Smaller models aren't always cheaper: the hidden costs LLM business cases miss

zeuslac · reddit · 2026-09-22

The author argues most AI business cases only track the model bill and miss where cheap-per-call models actually make workflows more expensive.

The mechanism:

On easy cases a small model works and the saving is real. On hard cases its suggestions are weak enough that reviewers check them against source documents and redo the work — slower than no AI at all.
Errors that slip through review cost far more to fix later than inference ever cost. The model bill drops while total work cost rises, and nobody notices because review time sits in a different system.

Two counterintuitive takeaways:

Difficulty-based routing's value depends entirely on the price gap; a single price drop can eliminate the reason for the routing layer, so keep it cheap to unwind.
A router can be financially better yet fail on quality: send enough hard cases to a weak model and the overall error rate can exceed the manual baseline even while the spreadsheet shows savings.

The fix is unglamorous: one record per task tying model usage, review time and later corrections together, so you see the full cost of a completed task rather than the cost of a call.

Original post →

Smaller models aren't always cheaper: the hidden costs LLM business cases miss

More from AGI Musings