2DSD Speculative Decoding Speeds Up Looped LMs up to 7.64x With Lower KV Memory
LucaAmb · x · 2026-11-20
Looped LMs add capability via depth rather than parameters, but each extra loop costs KV cache and per-loop latency. The team behind 2DSD (Two-Dimensional Speculative Decoding) combines speculation along both tokens and depth with cross-loop KV sharing.
- Up to 7.64x speedup over standard autoregressive decoding with lower KV memory
- Released a Hugging Face collection with shared-KV adaptations and drafter models for Ouro-1.4B/2.6B-Thinking, Nanbeige4.2-3B, and Dory1.0-4B
More from Infra
- US House Reveals 3.4M Chip Export Applications to China, Including 445,000 NVIDIA H200s — Dan_Jeffries1 · 2026-11-20
- Quantum Computing Takes Aim at AI Workloads as IBM and QuEra Map Fault-Tolerant Roadmaps — mikeflache · 2026-11-20
- Tencent Earnings Call: Latest Chip Restrictions Will Accelerate China's Domestic Semiconductor Supply Chain — basedjensen · 2026-11-20
- Chorus paper: shared n-gram tables cut memory 87% while beating DeepSeek's Engram — brown2green · 2026-11-20
- px0: A 20MB browser-based IDE for AI-generated code, built by chopping layers with first-principles thinking — arpit_bhayani · 2026-11-20
- Tsinghua-spinoff Qingyi launches Ascend 950DT 10,000-GPU token factory, lifting utilization from 40% to 80% — 量子位 · 2026-11-20