KLPO: a critic-free, single-rollout method for off-policy agentic RL, with an authorship riddle

lpachter · x · 2026-09-22

Yifan Zhang et al. released KLPO (KL-Regularized Policy Optimization), a critic-free, single-rollout approach to asynchronous off-policy agentic RL:

Core idea: fits a local policy mirror descent condition by regressing trainer-to-sampler log-ratios — no multiplicative importance weights, no critic, no reward groups; the default token-regression uses a per-token return coefficient plus score centering.
MC-KL: Monte Carlo estimation of the conditional score mean with M≥1 independent sampler draws per prefix recovers the full-KL gradient in expectation without extra response rollouts; a sequence-regression objective follows under deterministic token transitions and terminal rewards.
Engineering payoff: no prompt-and-version normalizers to estimate, so asynchronous replay needs no auxiliary normalization predictor or same-prompt response groups; TopK-KL and Binary KL are optional approximations.

Separately, LP Achter jokes that the "et al." in the author list is a riddle — could be one person, all 8.3 billion humans, a model, or a dog pressing the return key, poking at the report's opaque authorship.

Original post →

KLPO: a critic-free, single-rollout method for off-policy agentic RL, with an authorship riddle

More from Fun