Qwen MoE stability paper shows training-inference KL divergence rising over steps
heghbalz · x · 2026-07-24
- The post points to a Qwen MoE stability paper and highlights a plot of training-inference KL divergence.
- The author says they are trying to understand the intuition behind why the KL divergence between the trainer policy and inference grows over time, even though the experiments are synchronous and not due to async/off-policy effects.
- The attached chart shows KL divergence trending upward across gradient steps, with one run diverging sharply early.
More from Research
- Stanford Data-Centric AI Workshop reopens site, extends poster deadline — jure · 2026-10-26
- Hugging Face wraps 6-month Training Agents series with open recordings and code — SergioPaniego · 2026-10-26
- 'Vibe verification': ~60% of AI-written formal specs prove the wrong property — basedjensen · 2026-10-26
- Ito: a browser that detects browsing intent and auto-organizes tabs into Flows — andrewkuznet · 2026-10-26
- Cornell study says brains process language as linear sequences, challenging Chomsky's hierarchy — anirbanbandyo · 2026-10-26
- Hi3D V1.0 open-sourced with 2M training assets and 90% slicing success rate — kimmonismus · 2026-10-26