DeepMind bakes speculative decoding into Gemma 4: up to 3.1x server speedups, 2.2x on smartphone GPUs
dair_ai · x · 2026-11-22
dair-ai breaks down Google DeepMind's new paper on speculative decoding for Gemma 4. A shared multi-token prediction drafting head reads the backbone's activations and KV cache, removing the separate prefill step, plus a clustered vocabulary projection to cut draft cost.
Key results:
- Up to 3.1x speedups for large models on servers; 2.3x on H100 and 2.2x on smartphone GPUs for small models
- Domain adaptation: distilling the drafter alongside a fine-tuned backbone needs up to 1.83x fewer target forward passes than tuning the backbone alone
- Code for the adaptation recipes is open-sourced
Big deal for agents generating long outputs: lower latency and serving cost.
More from Infra
- Intel's 350W D270 claims 5.9x lead over RTX PRO 6000 — but the fine print reveals benchmark games — StefanoGogioso · 2026-11-22
- Inkling 2 Mini IQ4_NL hits 44 tok/s on an RTX 5060 Ti 16GB — llama.cpp params included — bobaburger · 2026-11-22
- Chollet: GPU spot prices keep rising as HBM sells out through 2029 — AI compute buildout isn't over — fchollet · 2026-11-22
- Gavin Baker: No investor feels misled by Nvidia's China GPU disclosures — GavinSBaker · 2026-11-22
- 'Urgent' power crisis hits Cambridge, Ontario after Cohere joins local data centre, bills rising — _ghnn_ · 2026-11-22
- Reporter confirms data center noise is real: like a plane overhead that never lands — jdjohnson · 2026-11-22