Baseten's Kiely: Speculative Decoding Is the Fastest-Moving Front in Inference
AI Engineer · youtube · 2026-09-20
Philip Kiely of Baseten reviews what changed in inference engineering since his book. The through-line: local inference is "get it working, then make it less dumb"; data center inference is "get it working, then make it less slow" — and the biggest shift is that data center optimizations increasingly come from dedicated training, blurring training and inference.
- Quantization: the data center answer stayed 4-bit weights, not a compressed cache. After TurboQuant went viral, his team did the math: 4-bit cache doubles effective bandwidth but cuts tokens/s by more than half — unacceptable in a data center, near-ideal on a memory-starved local machine.
- Caching: the frontier is compaction — a learned bottleneck cross-attends fixed query vectors against the full KV cache and emits compact keys/values in one forward pass.
- Speculation: moved fastest. Small same-family drafters were never good; training on the target's hidden states worked far better; the diffusion drafter (DFlash) proposes 8–16 tokens at once and more than tripled acceptance in production; DSpark pairs it with a sequential drafter; continuously retraining the speculator on live prompts lifts acceptance 20%–2x, if you can afford storage, compute, and permission.
More from Infra
- Nscale revenue up 1,252% yet loses over $1 billion in six months — TansuYegen · 2026-09-20
- HilbertRaum: open-source app packs local models, docs and chats onto a USB drive — Vladowski · 2026-09-20
- 500K Embedding Tokens/Sec on One GPU: Superlinked's Small-Model Serving Stack — AI Engineer · 2026-09-20
- Entity resolution with Jev cut pipeline costs 99.56% and boosted throughput 7.35x — hrishioa · 2026-09-20
- Home 8x RTX 6000 Pro rig now costs as much as a condo, warns builder — TheZachMueller · 2026-09-20
- SGLang x Datawhale Add New Chapters to Open-Source Inference Engine Course — ying11231 · 2026-09-20