Same model, 2.3x cost gap: cache hit rates make per-token pricing a poor guide
nicolechirps · x · 2026-10-24
ValsAI's benchmarks show price per token is a poor guide to real spend, because providers serve the same model with different cache hit rates. The clearest case: Kimi K3 on Baseten has near-identical per-token prices, yet a task cost $12.65 vs as little as $5.43 elsewhere — Baseten served only 72% of input from cache, versus 94% on Together AI and Fireworks, so more of each prompt was billed at full rate.
Takeaway: compare task-level costs and cache hit rates, not sticker prices.
More from Infra
- Google says capacity stays tight into 2027 as AI labs book AWS faster than it can build — tzmartin · 2026-10-24
- Semis roar back into VC focus as CASPA panel calls for 1000x faster, greener AI chips — hardimanjames · 2026-10-24
- Dev squeezes a 7M-param byte-level LLM entirely into an RTX 4090's on-chip memory — dejanseo · 2026-10-24
- Local LLM storage pain: an 8TB Gen5 SSD now costs $2,719.99 — TheZachMueller · 2026-10-24
- DeepSeek nears ~$15B round at $74B pre-money valuation, double its original target — 2C_ornot2C · 2026-10-24
- Carmen Li wants to turn AI compute into an investable asset class like oil — danielrock · 2026-10-24