KV cache billing dispute: realized-hit pricing may incentivize gaming, OpenRouter cited
charles_irl · x · 2026-10-29
cHHillee highlights a split in inference provider billing: most charge on realized KV cache hits, while Tinker bills on theoretical hits (1-hour TTL), closer to Anthropic's explicit TTL model for cache retention.
The author argues Tinker/Anthropic's approach makes more sense: under realized-hit pricing, inference engines that optimize for more cache hits actually reduce their own revenue — a misaligned incentive, with signs of such gaming already visible on OpenRouter.
The quoted post points to another direction: "an internet for the KV cache" — shipping a KV cache across the network can now be faster and cheaper than recomputing it.
More from Infra
- Running Computer-Use Agents on a Mac Mini via Screen Sharing Beats a Linux Box — willdepue · 2026-10-29
- Google building 'Frozen v2' custom chip for Gemini, slated for 2028 — The Information — zacharynado · 2026-10-29
- Rumored M5 Pro/Max MacBook Pro: 128GB RAM, 16TB storage for local AI models — LearnWithBishal · 2026-10-29
- NanoGPT Speedrun record shattered: 39.9s to 6.7s with retrieval-augmented n-gram hints — tatsu_hashimoto · 2026-10-29
- Cerebras WSE-3 die is ~57x an H100 die, so TDDB failure odds could make the chips depreciate fast — teortaxesTex · 2026-10-29
- Modular opens full MAX source: compiler, runtime, backends, royalty-free commercial use — ryanshrout · 2026-10-29