KLPO: a critic-free, single-rollout RL method for agentic LLMs, framed as 'Q* solved'
inductionheads · x · 2026-09-21
Yifan Zhang released KLPO (KL-Regularized Policy Optimization for Critic-Free Agentic RL), a paper and open-source repo announced with tongue-in-cheek framing like "Q has been solved" and "the grand finale of RL science."
The method itself is substantive:
- FlashREINFORCE / KLPO: a critic-free, single-rollout, asynchronous RL method for agentic language models, arguing RL should just do REINFORCE
- Default implementation: token regression + Monte Carlo KL (MC-KL) — terminal rewards provide feedback, and independent auxiliary token draws at visited prefixes estimate the sampler-conditioned score correction; no same-prompt response groups or learned value/normalizer models needed
- One complete response per prompt suffices for training.
Code, technical report and training guide are on GitHub (yifanzhang-pro/KLPO).
Related event: KLPO paper claims Q* solved, drawing community mockery(4 posts)→
More from Research
- Autonomous agent edits just 1,024 weights to lift Qwen3.5 0.8B from 7.4% to 37.4% on MATH-500 — VictorSanh · 2026-11-12
- Gait dataset labeled by encounter, spanning pre-op to 3 months post-TKA — zawthet · 2026-11-12
- Walking alignment measured 2.5-3.1° more neutral than X-ray, iPad gait study finds — zawthet · 2026-11-12
- MIT's Tegmark coins 'formal-washing': Signal protocol proof is far from complete — tegmark · 2026-11-12
- Signal formal verification hits 80%, Tegmark says Wired overstated completion — tegmark · 2026-11-12
- Category-theoretic framework asks: when does a physical system actually compute? — neurovium · 2026-11-12