Liquid Inference serves 10B tokens/day, 88.7% of inputs are cache reads
AccBalanced · x · 2026-10-28
Brett Harrison says Liquid Inference now serves 10B tokens daily, growing 200% week over week, mostly from agents and long chats. Across 470k billed jobs, 88.7% of input tokens were cache reads.
The key insight: in multi-turn/agentic workloads the biggest cost and latency wins come from prefix/KV cache reuse, but naive load balancing or pure price auctions scatter sessions across providers, destroying those savings. Their matching algorithm ranks offers by expected charge inclusive of cache hit rate rather than list price, routing sessions to providers that still hold the cached prefix.
More from Infra
- Will local-model progress outrun GPU depreciation? Reddit debates the buy-now window — Sensualities · 2026-10-28
- Dev cuts local AI inference power from 100W to 25W with only a 1.9s→2.8s slowdown — AryanEmbered · 2026-10-28
- SK Hynix posts $62.8B quarterly operating profit at 80% margin as AI memory demand booms — mhdfaran · 2026-10-28
- SK hynix plans ~KRW 100T shareholder returns in FY2026 on AI-driven memory boom — firstadopter · 2026-10-28
- SK hynix: Global CSP CapEx to grow 60%+ YoY, AI DC infra market at 40% CAGR through 2030 — firstadopter · 2026-10-28
- OpenAI plans to use Google's TPUs, The Information reports — zephyr_z9 · 2026-10-28