LoopFill: training-free speculative decoding speeds up looped Transformers

UIUC-CS · hf · 2026-11-17

UIUC researchers introduce LoopFill, a training-free speculative decoding algorithm for looped language models. Looped Transformers reuse the same layers repeatedly for parameter efficiency, but decoding requires T sequential passes per token, which is memory-bound at small batch sizes. LoopFill merges multi-token suffix drafts with shallow-loop predictions into a single draft tree with full-depth verification, preserving the autoregressive output distribution. It exploits the inductive bias that shallow-loop predictions closely match full-depth ones: on Ouro-2.6B (T=4), the top-1 token after one loop matches the full-depth greedy token at 88.8% of positions, rising to 95.6% after two loops.

Original post →

More from Infra

Infra channel →