Energy-Based Transformers learn to think via unsupervised energy minimization, scaling 35% faster than Transformer++
rbhar90 · x · 2026-09-06
An arXiv paper proposes Energy-Based Transformers (EBTs), a new class of Energy-Based Models that learn System 2-style thinking purely from unsupervised pretraining. EBTs assign an energy score to each input-candidate prediction pair, then reframe prediction as gradient descent-based energy minimization until convergence. Unlike existing inference-time compute methods, they need no verifiers or verifiable rewards and generalize across discrete (text) and continuous (visual) modalities. Across training, EBTs scale up to 35% faster than Transformer++ in terms of data, batch size, parameters, FLOPs, and depth.
More from Research
- JEPA may keep unpredictable features: new paper gives set-level characterization — hisspikeness · 2026-10-23
- GAFU Uses Generation-Side Representations to Boost Fine-Grained Visual Understanding in Unified Multimodal Models — Tengju Ru · 2026-10-23
- NTU Study: Learned Unmasking Policies Improve Speed-Accuracy Trade-offs in Diffusion Language Models — nanyang-technological-university-singapore · 2026-10-23
- NBER releases 'The Economics of Transformative AI' edited by Agrawal and Brynjolfsson — JMateosGarcia · 2026-10-23
- Swapping only the eval harness moves results 30%: designing agent evals like a scientist — AI Engineer · 2026-10-23
- Jane Street and NVIDIA speed up Mamba-2 with custom cuTile kernels — blelbach · 2026-10-23