NRGPT paper reimagines GPT inference as gradient descent on an explicit energy landscape
burkov · x · 2026-11-04
Andriy Burkov summarizes the NRGPT paper, which asks whether the standard GPT approach to text prediction can be expressed as an energy-based model, treating inference as movement across a landscape toward more plausible token representations. An explicit energy could make model behavior easier to analyze and enable more flexible inference and control.
- NRGPT is a GPT-like causal architecture that keeps next-token prediction, but derives repeated updates to token representations from gradients of a defined energy function, with token-level attention and feed-forward energies
- The authors evaluate on nested arithmetic (ListOps), Shakespeare text, and OpenWebText, comparing against baselines
More from Research
- zkSNARKs Prove a 400B Model Wrote a Reply Without Revealing Weights, at 70-95x Compute — rickasaurus · 2026-11-04
- Mistral Large 4 Report: Mid-Training Architecture Swap, Learnable RoPE, RL Agents Escaping Sandboxes — _AndrewZhao · 2026-11-04
- Alibaba's SPAR sales agent lifts conversion 15.8% in A/B test across 200k+ merchants — ycombinator · 2026-11-04
- ScienceRSI: Small expert models lift a 27B target by 6.7 points across seven benchmarks — jiqizhixin · 2026-11-04
- NYU CDS research: AdaJEPA world model keeps learning post-training, plus Anthropic introspection rebuttal — mengyer · 2026-11-04
- 132-Page Tutorial on Generative Models for Music Audio Lands on Hugging Face — amaai-lab · 2026-11-04