BPCO paper distills a stable PPO recipe for LLM RL, beating GRPO across scales

max_paperclips · x · 2026-09-03

Researchers introduce Best-Practice Critic Optimization (BPCO), a practical recipe for training critics reliably in LLM RL.

• Starts from a small model and an easy dataset, adding stabilization techniques one at a time with clear A/B tests and reasoning
• Ablated against vanilla PPO and GRPO, then scaled along three axes: bigger datasets, bigger models, and rubric-based rewards — BPCO wins in multiple settings
• The authors stress there is "no secret sauce, no novelty, just implementation details that matter," noting community implementations are full of hidden traps

Paper and code are public; the retweeter calls it a must-read for practitioners for its approach to doing science alone.

Original post →

More from Research

• Chollet: all AI converges to symbolic learning as 8-year paper finds implicit symbolic structure in LLMs — burny_tech · 2026-09-03
• New Preprint with Tetlock: How RL Scoring Rules Reshape LLM Forecasting Behavior — simonguozirui · 2026-09-03
• DeepLoop paper makes looped transformers scalable; rumor claims frontier models are 48 layers looped twice — StartupYou · 2026-09-03
• KAIST's Declarative Attention lets LLMs skip most KV cache reads — kaist-ai · 2026-09-03
• NVIDIA post-training pipeline hits gold-medal IOI performance, topping top humans — nvidia · 2026-09-03
• Kirin builds large-scale animal motion dataset from in-the-wild video for 3D animation — Brian Nlong Zhao · 2026-09-03

Research channel →