Latent feedback cuts Transformer training compute by up to 46% with 4 lines of code

IFM · hf · 2026-11-17

Latent Feedback Transformers feed each token's top-layer hidden state into the next token's input, acting as a compute and data amplifier. The change requires only 4 added lines in the training loop (a nograd forward of the previous state, rms-normed and added to embeddings). Compute savings scale with unique tokens per parameter: -8% at 12.5 TPP, 15% at 100, and 46% at 400. Unique data savings range from 10% (60M model) to 20% (778M). Scaling-law predictions: 34% compute saved at 10B parameters, 60% at 1,000 unique TPP, and 49% with an 8-epoch budget.

Original post →

More from Research

Research channel →