Same model, different speed: FriendliAI explains up to 7x inference gains from caching and speculation
AI Engineer · youtube · 2026-10-12
At AI Engineer World's Fair, FriendliAI founding engineer Yunmo Koo argues open-weight models like GLM 5.2 now match proprietary ones (5.6x cheaper than Opus on a tower-defense demo) and the real gap is the inference provider. He details prefix caching with cache-aware routing, a GPU/CPU/NVMe cache hierarchy for long repetitive agent inputs, and hybrid speculative decoding that adapts to live traffic; reliability counts as part of speed since one failed call can sink an agent run. Kilo Code saw up to 7x faster responses with fewer errors.
More from Infra
- Rumor: New Architectures Could Cut Cerebras K3 Wafer Needs From 60 to 10 — bookwormengr · 2026-10-23
- Nvidia plunges 7.21% to $212.31 in a day, still holding a $5.2T market cap — michalmalewicz · 2026-10-23
- Serverless Is a Billing Model: How to Price Stateful Compute by Usage — s4chinraja · 2026-10-23
- AKV Serving System Lets Agents Manage Their Own KV Cache, Cutting Cost 27.7% — _akhaliq · 2026-10-23
- Custom AI chips from AWS and Google are the new cloud lock-in — DavidLinthicum · 2026-10-23
- AI helps improve Linux's AMD Zen CPU microarchitecture detection in the kernel — Fcking_Chuck · 2026-10-23