Dev audits KLPO training backend: default gradient equals REINFORCE, async training rejected

burny_tech · x · 2026-09-21

A developer inspected the KLPO project's loss implementation and training backend, ran all 145 tests, and verified the gradients by hand.
Key finding: when trainer and sampler probabilities match, the default gradient equals REINFORCE in expectation (numerically reproduced); the correction only matters when they diverge.
The backend mandates synchronous, on-policy training with one optimizer step per rollout batch and explicitly rejects asynchronous configurations, substantially limiting continual-learning potential.
Its default Monte Carlo implementation retrieves the full vocabulary's probabilities at every token and samples auxiliary tokens with NumPy, adding overhead without cheaper training; single-trajectory updates already exist in REINFORCE and don't establish reliable learning.

More from coding & agent