ArgMaxRL: unbiased gradient estimator jointly optimizes all best@k objectives to fix RL diversity collapse
fpedregosa · x · 2026-09-02
Standard post-training RL objectives like GRPO optimize the expected quality of a single sample, causing mode collapse: greedy accuracy rises but the model can't produce diverse high-quality outputs, wasting test-time scaling potential.
ArgMaxRL, a new gradient estimator building on MaxRL and generalizing it from binary to continuous rewards:
- Jointly optimizes all best@k objectives (expected quality of the best output among k samples) for every k up to the sampling budget, weighted as Σ 1/k·best@k(x)
- Provides an unbiased, closed-form gradient estimator — a few lines of code that drop into existing training loops
- Recovers binary MaxRL exactly as a special case
The motivation: agentic systems tackling expert-level problems care about best@k, not top-1, and this work bakes test-time scaling directly into the training objective.
More from Models
- Tencent open-sources WeMM-Embedding: unified text/image/video embeddings, Apache 2.0 — tomaarsen · 2026-09-02
- When nobody can track frontier model progress, closed-model business may lose to open weights — StewartalsopIII · 2026-09-02
- Looped transformer is no dark art: rasbt debunks the OpenAI Astra rumor — rasbt · 2026-09-02
- GLM 5.2 slug references spotted in Google Antigravity CLI, hinting at integration — gaganghotra_ · 2026-09-02
- Anthropic's Fable-5.1-max Grabs #4 on eyebench-v3 in Biggest Bench Jump Yet — adonis_singh · 2026-09-02
- Dev Tries Claude for Coding Agents Again, Finds Sol 5.6 Sharper But Less Friendly — iskander · 2026-09-02