Full-bandwidth Transformer Matches 1.5x Data at 1B Scale with <1% Overhead
ProfBuehlerMIT · x · 2026-08-14
The Full-bandwidth Transformer addresses the information bottleneck in autoregressive models, where typically only the sampled token is passed to the next step while the top-layer hidden state is discarded.
- Core Mechanism: It fuses the previous top-layer hidden state with the current token embedding via a gated linear unit, feeding it back to the bottom. This allows non-verbalized computation to re-enter the stack.
- Training Strategy: Uses a scheduled multi-pass objective introducing latent feedback late in pretraining, maintaining the standard architecture and KV cache.
- Results: At a 1B parameter scale trained on 400B tokens, it improves validation loss, math/coding generation, and instruction-tuned performance. With negligible (<1%) decoding overhead, it matches standard transformers trained on 1.5x more data and produces shorter reasoning traces.
More from Research
- Terence Tao Uses AI to Prove Sendov's Conjecture, Lean Formalization Completed — stevenstrogatz · 2026-08-14
- ARC Benchmark: Frontier Models Absorbing Harness Patterns for Better Reasoning — mhmazur · 2026-08-14
- Embodied AI Blind Spot: 9 VLMs Know How to Repair but Fail to Proactively Spot Issues — rohanpaul_ai · 2026-08-14
- Anthropic Researcher: Lean Kernel to Be Bulletproof by AI Within a Year — geoffreyirving · 2026-08-14
- Kevin Murphy's Solo Paper: Using LLMs and Bayesian Methods for Efficient Scientific Discovery — burny_tech · 2026-08-14
- Compute with Human Brain Cells: Parasma CEO Proposes Radical Energy-Efficient Alternative to Silicon — bosmeny · 2026-08-14