Late Entry in looped LLMs delivers up to 12.5x lossy and 8.4x exact speedups

NanyangTechnologicalUniversity · hf · 2026-11-19

NTU researchers propose Late Entry decoding for looped language models: new tokens skip initial passes and enter directly into the final K<T passes over a deep KV cache. A single static CUDA graph unifies greedy-exact (τ=0), margin-gated, and lossy verification. On Ouro-2.6B, K=2 Wake reaches 5.2x greedy-exact speedup (+26% on GSM8K); on Huginn-3.5B with T up to 64, speedups hit 8.4x exact and 10.6–12.5x lossy.

Original post →

More from Infra

Infra channel →