Late Entry in looped LLMs delivers up to 12.5x lossy and 8.4x exact speedups
NanyangTechnologicalUniversity · hf · 2026-11-19
NTU researchers propose Late Entry decoding for looped language models: new tokens skip initial passes and enter directly into the final K<T passes over a deep KV cache. A single static CUDA graph unifies greedy-exact (τ=0), margin-gated, and lossy verification. On Ouro-2.6B, K=2 Wake reaches 5.2x greedy-exact speedup (+26% on GSM8K); on Huginn-3.5B with T up to 64, speedups hit 8.4x exact and 10.6–12.5x lossy.
More from Infra
- CoreWeave Insolvency Speculation: $24.6B Debt Due vs $3.7B Quarterly Revenue — SumitGup · 2026-11-19
- Anthropic priced its IPO at $83 above range; anon thread predicts the stock halves before unlocks — basedjensen · 2026-11-19
- Satya Nadella says Microsoft would lease SpaceX compute for data centers in space — elonmusk · 2026-11-19
- AMD ROCm 10.2 Ships With 18 New Ryzen Laptop Models Supported for Local AI — AnushElangovan · 2026-11-19
- Consumer NAND spot prices drop 20% in a month, more SSD relief expected in early 2027 — SumitGup · 2026-11-19
- Tideway: phase-aware unified memory pool speeds MoE inference on consumer devices by ~2x — DukeUniversity · 2026-11-19