KL penalty fails in video RL: FlowGRPO stabilizes KL yet outputs stay noisy
gowthami_s · x · 2026-10-15
Part 2 of a thread on the DDRL paper: even with a stronger KL penalty, FlowGRPO's KL settles while generated videos still show noisy textures. The paper hypothesizes that the penalty compares models on states visited by the current policy; once those drift off-distribution, the frozen reference becomes unreliable too — stable KL, visibly worse generations. The author notes DDRL (keeping the diffusion denoising loss during RL) is an important regularization against reward hacking, verified by their own practice to stabilize the loss landscape.
Related event: DDRL keeps diffusion loss during RL to fix reward hacking in video models(5 posts)→
More from Multimodal
- Customuse launches Smart Mesh AI with geometry-aware adaptive character clothing tool — SimplyAnnisa · 2026-10-23
- AI-generated movie poster 'The Chase' crams a wolf, a plane and muscle cars into one chase — Aggravating-Sir-115 · 2026-10-23
- SteerTTS-edge: an 11.84MB steerable TTS with unchanged WER running 19x realtime on CPU — Working_Hat5120 · 2026-10-23
- ElevenLabs' Chimera pairs Eleven v4 voice with Seedance video to make character images speak — AIwithGhotai · 2026-10-23
- Open-Source Tennis Models Plus Claude Turn Video Into a World-Class Coach — tweetsatpreet · 2026-10-23
- Berkeley's StyleStream 2.0 cuts streaming voice conversion latency to 520ms, 2x faster — Berkeley · 2026-10-23