A sequence-level KL estimator can give the right gradient for async RL
ZacharyHuang12 · x · 2026-07-26
A technical thread argues that RL should be optimized at the sequence level, but most practical losses still collapse to token-level updates. The post highlights a paper that derives a sequence-level KL estimator with the correct gradient—which matters more than matching the exact KL value—and then extends the derivation to async RL.
Key point:
- The author shows how to apply importance sampling correctly for the sequence-level estimator.
- Although sequence-level IS is high-variance because it multiplies over all tokens, the derivation shows that only the future tokens relative to the current position contribute to the update.
- The conclusion is that the importance-sampling correction should be applied specifically to the derived sequence-level return term, not naively to every token.
More from coding & agent
- Claude + Kit MCP: Automating Email Marketing Workflows via Natural Language — evielync · 2026-07-26
- Builder replaces hundreds of installed agent skills with one registry-backed skill — abhimanyu_saharan · 2026-07-26
- Anthropic says its internal Slack agent now writes 65% of product team code — VraserX · 2026-07-26
- Aethos Memory lets Claude Code, Cursor, and Windsurf share one persistent brain — Longjumping-Koala396 · 2026-07-26
- Open-sourced MCP memory server for coding agents uses 3-pass RAG retrieval — Longjumping-Koala396 · 2026-07-26
- LLM IMO 2026 test shows harnesses help, but frontier models still stay ahead — pequalnp92 · 2026-07-26