Feedback Shortcut: 2.4–2.8x faster speculative decoding for latent-feedback Transformers

JohnCLangford · x · 2026-11-21

A new paper fixes the incompatibility between latent-feedback Transformers and speculative decoding. Feeding each position's top-layer hidden state into the next position boosts quality but breaks parallel draft verification, since a draft needs the preceding position's state.

Key ideas:

Results: 2.4–2.8x decoding speedup at 135M and 780M scale with 2% extra training compute, staying within 1e-4 bits/byte of the exact-decoding ceiling on Paloma. At large batch sizes, skipping correction keeps Transformer-level speed with over 80% of latent feedback's gains. Paper and code are available.

Original post →

More from Infra

Infra channel →