DLite: History-Free Speculative Decoding Fixes 80% Acceptance Drop in Agentic AI
SJTU · hf · 2026-10-19
DLite, from SJTU, Shanghai AI Innovation Institute, Tsinghua and Infinigence-AI, tackles speculative decoding under agentic workloads: long, continuously evolving contexts and frequent multi-turn interactions cause up to an 80% degradation in acceptance length for existing draft models, while maintaining historical KV cache adds substantial system overhead. DLite is a history-free framework that directly reuses contextual hidden states from the target model for local speculative prediction without explicitly maintaining history, boosting acceptance length and reducing overhead.
More from Infra
- Sipeed's MaixStation challenges DGX Spark: 128GB LPDDR5X at 40W, one-third the size — AIFlow_ML · 2026-10-19
- ReaDy Uses Recurrent States as Dynamic Context for Speculative Decoding in Hybrid Linear Attention LLMs — Bohan Sun · 2026-10-19
- TRELLIS.2 image-to-3D running on AMD RX 9060 XT with ROCm: fixes, workflows, MIT swaps — ova-mojica · 2026-10-19
- Celesto Claims CI Runner With 96% Lower Per-Minute Cost Than GitHub Actions — aniketmaurya · 2026-10-19
- Huawei's Atlas 960E superpod links 4,096 NPUs with near-packaged optics — bookwormengr · 2026-10-19
- Robot brain inference cost fell 84% in two months, from $2.36 to $0.38 per task — victor_explore · 2026-10-19