Matryoshka: Speculative Decoding Speeds Up Inference by 10-30%
nthngdy · x · 2026-08-19
At inference time, speculative decoding is performed on the largest model using smaller submodels as drafts. Nesting allows shared memory and activations, amortizing draft overhead. This achieves 10-30% faster decoding. During training, all submodels are trained together, enabling free student-teacher online distillation during pretraining for better alignment.
Related event: Matryoshka LM Suites: Nested Training Cuts Compute by 36%(8 posts)→
More from Infra
- CUHK Team Open Sources Libra: 3x Throughput for Agentic Training — jiqizhixin · 2026-08-21
- Open-source x402-cleanweb-agent saves 80% tokens by cleaning web content — EstablishmentTough18 · 2026-08-21
- Pretraining Potential: Coding Agents and the Compute Bottleneck — zeeshanp_ · 2026-08-21
- The Math: Claiming 100T Tokens/Day Would Need ~580K GPUs — teortaxesTex · 2026-08-21
- Moore's Law Fading: Non-Silicon Computing and Novel Architectures to See Capital Influx — MikePFrank · 2026-08-21
- "Why do we need more datacenters? Just write faster kernels" — basedjensen · 2026-08-21