LoopFill: training-free speculative decoding speeds up looped Transformers
UIUC-CS · hf · 2026-11-17
UIUC researchers introduce LoopFill, a training-free speculative decoding algorithm for looped language models. Looped Transformers reuse the same layers repeatedly for parameter efficiency, but decoding requires T sequential passes per token, which is memory-bound at small batch sizes. LoopFill merges multi-token suffix drafts with shallow-loop predictions into a single draft tree with full-depth verification, preserving the autoregressive output distribution. It exploits the inductive bias that shallow-loop predictions closely match full-depth ones: on Ouro-2.6B (T=4), the top-1 token after one loop matches the full-depth greedy token at 88.8% of positions, rising to 95.6% after two loops.
More from Infra
- Pipeshift claims 40% faster first token than Sarvam serving Gemma 4 31B at 150B tokens/day — NirantK · 2026-11-17
- Tencent's DSpine speculative decoding ships GGUFs and a llama.cpp PR: 4.5x speedup on 27B — pmttyji · 2026-11-17
- Founder lands 1000 GPUs (40% GB300, 60% B300) on short-term contracts, dismissing the GPU shortage narrative — SumitGup · 2026-11-17
- Agent Substrate: open-source zero-trust runtime suspends idle agents, fits 10x more per hardware — Saboo_Shubham_ · 2026-11-17
- Practical tips to cut agent memory costs on OpenAI: cache TTL, Flex, fewer calls — gabrielchua · 2026-11-17
- xAI affiliate buys 96 acres near Memphis Colossus 2 for $5.5M as compute footprint expands — XFreeze · 2026-11-17