New paper pins RLVR entropy collapse on sampler mismatch, not black magic
joecole · x · 2026-11-02
An arXiv paper ("A Sign Law for Entropy Drift under Sampler Mismatch in RLVR") explains why post-trained reasoning models see entropy randomly collapse or blow up.
- Key finding: since nearly everyone samples with top-k/top-p/min-p or tweaked temperature rather than the raw policy, sampler mismatch makes constant advantage shifts cause systematic entropy drift, with a sign fixed by the sampler.
- The classic gotcha is explained: masking over-length responses crashes entropy, while marking them as failures blows it up—opposite directions.
- Token-level importance ratios account for 98% of the drift; four clipped minibatch updates amplify per-batch deviation sevenfold; a per-zero baseline only delays entropy explosion over 100 steps.
- Fixes: canceling first-order drift each step bounds entropy without changing the learner; on a 9B reasoning model, zeroing net advantage or renormalizing the learner raised held-out accuracy by 8.2 and 8.8 points within ten steps.
More from Research
- HalachaBench sparks debate: is Talmudic Q&A testing knowledge or reasoning? — teortaxesTex · 2026-11-02
- Sebastian Raschka releases free 11-hour course on building reasoning models from scratch — AxSaucedo · 2026-11-02
- Microsoft and Berkeley open-source TokenEconomizer, cutting coding agent costs 18–31% — victor_explore · 2026-11-02
- TLA-Verus uses LLM agents to prove temporal properties of TLA+ specs — tianyin_xu · 2026-11-02
- J-Lens is just a first-order Taylor expansion, researcher points out — burny_tech · 2026-11-02
- Mollick: expert-level work benchmarks are brutally hard, and ARC-AGI, GDPval, METR are saturated — emollick · 2026-11-02