PPO Collapses LLM Output Distributions While GFlowNet Objectives Keep Them Uniform
kastnerkyle · x · 2026-10-15
A technique thread argues that distribution collapse in RL fine-tuning stems from the loss, not gradient descent. In a number-generation experiment: base model hits 50.5% valid samples, PPO fine-tuning reaches 95.8% but sharply concentrates probability mass, while a GFlowNet objective achieves 100% valid samples with a perfectly uniform distribution. Swap PPO for GFlowNet if you want diversity instead of mode collapse.
More from Research
- STEMBench launches: 6 STEM domains, zero full-target successes across all runs — mariofilhoml · 2026-10-15
- OpenAI gives $5M to Simons Foundation's $15M push to help mathematicians navigate AI-era results — OpenAI · 2026-10-15
- DeepPhylo infers phylogenetic trees without unrealistic evolutionary assumptions — anshulkundaje · 2026-10-15
- Wander preprint explores graph foundation models via shared objective and random-walk interface — mmbronstein · 2026-10-15
- CHI Reviewer: AI Slop Is Real, but Unusually Rigorous Papers Still Getting Through — IanArawjo · 2026-10-15
- Into the Unknown: CMU benchmark finds 114 of 135 AI-written research papers fail integrity checks — arankomatsuzaki · 2026-10-15