RL training stability tricks: prompt-mean loss, asymmetric clipping and entropy-adaptive GRPO
tokenbender · x · 2026-09-22
The author breaks down key RL training modifications: sticking with GRPO rather than fancier algorithms but tuning it for stability and scale — prompt-mean loss over token-mean loss to stop long answers from getting excess gradient weight (DAPO); separate clipping bounds for positive/negative advantages (DAPO); entropy-adaptive clipping (mai-thinking-1); and segment-level credit shaping (TreeAdv).
They also note that 13% policy-drift staleness with large throughput implies the chosen environments run or improve directly on base weights without any curriculum — otherwise training would have suffered penalties.
Related event: Xiaomi MiMo's 1M-Context RL Training Details Spark Debate(13 posts)→
More from Models
- Alibaba Roadmap: 20GW Global Compute by 2032, Zhenwu Chips 3x Faster — kevinsxu · 2026-09-22
- Experiment suggests modern LLMs like Qwen hide tiny GPT2 self-models inside — paraschopra · 2026-09-22
- A watermarked real photo got tagged "Made with AI", exposing detection flaws — shashib · 2026-09-22
- Xiaomi MiMo-V2.6 details its largest RL scaling run: $2.6M, 1M-token contexts — KyeGomezB · 2026-09-22
- Hands-on: testing Grok 4.7 coding in Cursor across 4 real projects with cost breakdown — Arindam_1729 · 2026-09-22
- Xiaomi's MiMo v2.6 tops open-model index, RL run cost ~$3.5M and was livestreamed — Prompt Engineering · 2026-09-22