MaxRL: a one-line change turns RL into maximum likelihood, up to 20x more test-time compute-efficient than GRPO
burny_tech · x · 2026-10-25
- An ICML 2026 oral paper (arXiv:2602.02710) argues that when feedback is terminal and binary, standard expected-reward RLVR is only a first-order approximation of the true objective: the likelihood of generating correct solutions.
- The authors propose MaxRL, expressing maximum-likelihood gradients as a weighted combination of pass@k objectives. The practical change is nearly trivial: normalize successful trajectory gradients by the number of successes rather than total samples. The estimator is unbiased for a truncated likelihood objective and approaches ML optimization as sampling compute scales.
- MaxRL Pareto-dominates existing methods across all tested models and tasks, improves solution coverage on Qwen3 math reasoning while keeping competitive pass@1, achieves up to 20x better test-time scaling efficiency than GRPO, and scales more favorably with data and compute.
More from Research
- EMNLP paper introduces Concept-Targeted Attribution, tracing the circuits behind linear probes — ZhijingJin · 2026-10-25
- bbtree claims fastest concurrent btree: 5x faster than std BTreeMap single-threaded — tianyin_xu · 2026-10-25
- Covert channel math: 128-bit secret leaks in 13s, 1TB weights take centuries — vishalmisra · 2026-10-25
- CultureScore, a cultural evaluation benchmark, to be presented at EMNLP 2026 — pliang279 · 2026-10-25
- Why Formal Verification Is Exploding Now: Proofs Get Cheaper, Exploits Costlier — burny_tech · 2026-10-25
- Residual-stream ontologizer clusters 1,711 EA Global attendees into 48 types — amplifiedamp · 2026-10-25