Transformers Effectively Compute Amortized Gradients of a Non-Conservative Field
teortaxesTex · x · 2026-08-03
Amidst a discussion on whether explicit gradients should be used at test-time, developer doomslide offered a deep theoretical perspective:
- Reinterpreting Transformers: The forward pass of a Transformer is effectively computing the amortized gradient of a "fake" Energy-Based Model (EBM).
- Non-Conservative Nature: This gradient field is characterized as non-conservative.
- Context: This insight responds to a query by willdepue, who questioned why gradients—a ridiculously powerful tool—are exclusively used during training, noting the apparent absence of explicit gradients at test-time as a potential flaw.
More from Research
- Applying Mechanistic Interpretability to Video Models: Exploring Latent Space Physics Simulators — mathemagic1an · 2026-08-03
- NBER Paper: Automation Strips Work of Meaning, Potentially Harming Workers — joshgans · 2026-08-03
- Matrix Factorization Approach for Dynamic Rank/Select in Data Streams — minilek · 2026-08-03
- Explorative Modeling: A Third Pretraining Axis Beyond Parameters and Data — PMinervini · 2026-08-03
- HOST: Open-Source Robot Learns New Skills in 29s from Single Video, 507x Faster Than SFT — chris_j_paxton · 2026-08-03
- Surge AI Exec Reveals Why Benchmarks Fail: $15M Cost and Widespread Contamination — alex_verem · 2026-08-03