KLPO: a critic-free, single-rollout RL method for agentic LLMs, framed as 'Q* solved'
inductionheads · x · 2026-09-21
Yifan Zhang released KLPO (KL-Regularized Policy Optimization for Critic-Free Agentic RL), a paper and open-source repo announced with tongue-in-cheek framing like "Q has been solved" and "the grand finale of RL science."
The method itself is substantive:
- FlashREINFORCE / KLPO: a critic-free, single-rollout, asynchronous RL method for agentic language models, arguing RL should just do REINFORCE
- Default implementation: token regression + Monte Carlo KL (MC-KL) — terminal rewards provide feedback, and independent auxiliary token draws at visited prefixes estimate the sampler-conditioned score correction; no same-prompt response groups or learned value/normalizer models needed
- One complete response per prompt suffices for training.
Code, technical report and training guide are on GitHub (yifanzhang-pro/KLPO).
More from Research
- mini Jev: an open 1.05MB decision head on frozen Qwen3 0.6B outputs agent action probabilities — AlternativeSure2891 · 2026-09-21
- Weighting others by relatedness isn't new: paper models agents learning relational value — xuanalogue · 2026-09-21
- Paper critique: MLP size unaccounted for in Transformer logarithmic depth bounds — kfountou · 2026-09-21
- Robot Leaderboard Under Fire: 5 Trials Per Task, Authors Commit to 15 — YuXiang_IRVL · 2026-09-21
- Tabular foundation models enter the arena, mocked: 'wait until you find out about xgboost' — burny_tech · 2026-09-21
- LLMs can draw DAGs but can't guarantee d-separation, notebook discussion — fdellaert · 2026-09-21