Feedback Shortcut: 2.4–2.8x faster speculative decoding for latent-feedback Transformers
JohnCLangford · x · 2026-11-21
A new paper fixes the incompatibility between latent-feedback Transformers and speculative decoding. Feeding each position's top-layer hidden state into the next position boosts quality but breaks parallel draft verification, since a draft needs the preceding position's state.
Key ideas:
- State alignment: a lightweight projection from multi-token prediction head states creates stand-ins for the missing feedback hprev, leaving base training untouched.
- Pipelined correction: drafts are re-evaluated with their predecessor's newly computed state inside the next verification pass — better accuracy with zero extra calls.
Results: 2.4–2.8x decoding speedup at 135M and 780M scale with 2% extra training compute, staying within 1e-4 bits/byte of the exact-decoding ceiling on Paloma. At large batch sizes, skipping correction keeps Transformer-level speed with over 80% of latent feedback's gains. Paper and code are available.
More from Infra
- Altimeter breaks down the AI supercycle: nearly $860B annual CapEx and where it's going — bobmcgrewai · 2026-11-21
- AWS publishes SOSP paper on validating S3's distributed metadata index with lightweight formal methods — blaizedsouza · 2026-11-21
- Unsloth release runs 26.4B Inkling 2 Mini on 16GB RAM, cuts MCP token use 87% — danielhanchen · 2026-11-21
- Seroter Daily Reading List #914: harnesses are cheap to swap, model behavior isn't — rseroter · 2026-11-21
- NVFP4 MoE experts force CPU fallback in llama.cpp; requantizing boosts Metal speed 5.8x — teortaxesTex · 2026-11-21
- Compute overbuild skeptic recants: efficiency gains cut AI compute cost ~4x per year — teortaxesTex · 2026-11-21