Baseten's Kiely: Speculative Decoding Is the Fastest-Moving Front in Inference
AI Engineer · youtube · 2026-09-20
Philip Kiely of Baseten reviews what changed in inference engineering since his book. The through-line: local inference is "get it working, then make it less dumb"; data center inference is "get it working, then make it less slow" — and the biggest shift is that data center optimizations increasingly come from dedicated training, blurring training and inference.
- Quantization: the data center answer stayed 4-bit weights, not a compressed cache. After TurboQuant went viral, his team did the math: 4-bit cache doubles effective bandwidth but cuts tokens/s by more than half — unacceptable in a data center, near-ideal on a memory-starved local machine.
- Caching: the frontier is compaction — a learned bottleneck cross-attends fixed query vectors against the full KV cache and emits compact keys/values in one forward pass.
- Speculation: moved fastest. Small same-family drafters were never good; training on the target's hidden states worked far better; the diffusion drafter (DFlash) proposes 8–16 tokens at once and more than tripled acceptance in production; DSpark pairs it with a sequential drafter; continuously retraining the speculator on live prompts lifts acceptance 20%–2x, if you can afford storage, compute, and permission.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04