Stop-gradient on sampling yields a principled but high-variance update direction

mblondel_ml · x · 2026-11-17

Part three of Blondel et al.'s paper thread: applying a stop gradient to the sampling procedure yields a theoretically principled "moment matching" update direction, but it suffers from high variance in practice. See the thread's first post for the full analysis.

Related event: DeepMind paper proves sequence-level supervised losses for autoregressive models can be fundamentally flawed(5 posts)→

Original post →

More from Research

Research channel →