Paper proposes Full-bandwidth Transformer with latent feedback mechanism
burny_tech · x · 2026-08-15
The paper introduces the 'full-bandwidth transformer,' addressing the bottleneck in the vertical feedback channel of autoregressive transformers. While dense attention allows broad horizontal access, standard architectures only feed the sampled token back to the bottom, discarding the top-layer hidden state. The proposed 'latent feedback' mechanism fuses the previous top-layer hidden state with the sampled token embedding to serve as the next input, allowing non-verbalized computation to re-enter the stack with renewed depth. Training uses a scheduled multi-pass objective, introducing latent feedback late in pretraining for stability. Experiments on 1B-parameter models trained up to 400B tokens show improved validation loss, math/coding generation, and instruction-tuning performance, matching standard transformers trained with roughly 1.5x more tokens.
More from Research
- OpenMed Open Source Solution Solves Medical Data Privacy Challenges — aigclink · 2026-08-15
- Paper proposes efficient approximation for KL divergence between discrete normal distributions — FrnkNlsn · 2026-08-15
- RL Conference 2026 to focus on agents and self-improvement — tw_killian · 2026-08-15
- Fall Detection System Using Pose Estimation and Deep Learning — rsasaki0109 · 2026-08-15
- Study reveals hybrid LLMs organize computation around expensive full-attention layers — burny_tech · 2026-08-15
- New Framework Enables Multimodal Control for Humanoid Robots via Video and Music — zhengyiluo · 2026-08-15