arXiv paper gives StopGrads theoretical grounding: convergence without an underlying loss
srush_nlp · x · 2026-10-23
An arXiv paper by Max W. Shen, Mark Goldstein, Rajesh Ranganath et al. systematically studies stopgrads — ubiquitous in ML training but rarely analyzed.
Problem: stopgrads alter gradients, stationary points, and convergence guarantees, leaving training theoretically ungrounded — parameters follow a vector field that may not be the gradient of any loss.
Contributions:
- A "stopgrad regression principle" providing a general template with closed-form characterization of stationary points and uniqueness, unifying stopgrad objectives across flow maps, RL, and diffusion samplers
- Proof that the unique stationary point of stopgrad flow map objectives is the true flow map, with positive convergence results for Eulerian and Lagrangian objectives including MeanFlow and improved MeanFlow
- Under functional semi-gradient flow, the learned flow map has a closed-form expression composing the initial and true flow maps
- Modified stopgrad placements that cut training memory by 2x
The person who got scooped (Assaf Shocher) publicly endorses it: the authors explicitly solved idealized training dynamics and showed convergence to the right solution despite no underlying loss.
More from Research
- CMU PhD talk: patching full-attention transformers into hybrid architectures late in training — Cohere · 2026-10-23
- UCSC's ALPS encoder unifies state, language and pixels for world models in one ViT-B — yuyinzhou_cs · 2026-10-23
- Ultrafast waves flush brain waste in minutes: Science Advances mouse study on Alzheimer's — NickPassig · 2026-10-23
- PEARL demo: an LLM agent that checks experiment results, revises plans, and re-tests — KevinKaichuang · 2026-10-23
- Berkeley's StyleStream 2.0 cuts streaming voice conversion latency to 520ms, 2x faster — Berkeley · 2026-10-23
- Math PhD student quits academia: AI is making math unrecognizable — pickover · 2026-10-23