Prompt caching kills easy/hard model routing: cached big model now cheaper than small one
Lance_Saul_85 · reddit · 2026-10-14
A team's router sent easy requests to a small model and hard ones to a big model, lowering blended costs for months. Then prompt caching flipped the math: their 18k-token prefix caches on the big model at 1/10th input price (1.1¢/request), while the small model sees bursty traffic, misses cache, and pays full price (2¢/request). The small model now costs more on most requests and drags down the big model's hit rate. It only wins on long-output requests. They're leaning toward routing on expected output length instead of difficulty.
More from coding & agent
- TV noise crashes voice agents from 38.7% to 8.6%; full duplex is the fix — SIGKITTEN · 2026-10-14
- TypeSafe AI's Jev Claims 100x Faster Decisions, Can Replace 9 LLM Call Types — Arindam_1729 · 2026-10-14
- Gemini 4 Argon High tested with 'Horse prompt' via OpenCode V2 and Antigravity agent — ZealousidealTown1974 · 2026-10-14
- Buoy benchmark: Claude runs apps 6x faster and cheaper with MCP tools than none — LovesWorkin · 2026-10-14
- LMArena Launches Code Arena for WebDev Model Benchmarking — arena · 2026-10-14
- msgblast: Open-Source Mac App to Prompt Multiple AI Agents at Once — msg · 2026-10-14