Energy-Based Transformers learn to think via unsupervised energy minimization, scaling 35% faster than Transformer++
rbhar90 · x · 2026-09-06
An arXiv paper proposes Energy-Based Transformers (EBTs), a new class of Energy-Based Models that learn System 2-style thinking purely from unsupervised pretraining. EBTs assign an energy score to each input-candidate prediction pair, then reframe prediction as gradient descent-based energy minimization until convergence. Unlike existing inference-time compute methods, they need no verifiers or verifiable rewards and generalize across discrete (text) and continuous (visual) modalities. Across training, EBTs scale up to 35% faster than Transformer++ in terms of data, batch size, parameters, FLOPs, and depth.
More from Research
- Boltz Lab reveals 50x molecular token growth in 5 months as Series A closes — GabriCorso · 2026-11-19
- Researcher calls METR/Anthropic '2x speedup' RSI analysis statistically dubious — jessi_cata · 2026-11-19
- Linear Probes Could Reveal Which Persona a Base Model Is Playing, Researchers Suggest — cephaloform · 2026-11-19
- ETH's Anymal robot completes 10km autonomous alpine hike with VLM-guided ROAM system — ChongZzZhang · 2026-11-19
- OpenVax codebase grows from 78K to 565K lines in 15 months — iskander · 2026-11-19
- François Fleuret: decoder transformer's elegance — no trade-off hyperparameters — francoisfleuret · 2026-11-19