Reverse-engineering Tinker's StableReinforce: it's Reinforce + Score Centering
RichmanRonald · x · 2026-10-26
Tinker's new StableReinforce objective is revealed to be Reinforce + Score Centering, and the paper's author (Martin Marek) published a reverse-engineering thread showing how he figured it out.
The paper "Score Centering Stabilizes Off-policy Reinforcement Learning" (Together AI) shows:
- LLM RL is notoriously sensitive to training-inference mismatch (TIM); fully eliminating TIM is impractical due to rollout efficiency costs
- Instability under TIM is primarily caused by drift, a persistent bias that accumulates with every training step
- An additive "score centering" correction cancels drift, matching or beating importance-sampling methods at 0.6B–30B scale under quantization, with the gap growing as mismatch worsens; it also composes with importance sampling
- Experiments train Qwen3-30B-A3B-Base on INTELLECT-2 math data under FP8/FP4/INT8 quantization; score centering stays stable where PG, TIS, MIS, DAPO, GSPO and others collapse under heavy quantization
More from Research
- Real human brain cells are being used for AI, and Reddit is buzzing — EmotionalGuess9229 · 2026-10-31
- LLM judges harbor systematic errors that silently corrupt model rankings and training pipelines — burkov · 2026-10-31
- New paper quantifies ineffable concepts: 65 of 1,270 can never be taught to LLMs via language — sanmikoyejo · 2026-10-31
- Unsupervised Method Measures Whether LLM Messages Are Received — sanmikoyejo · 2026-10-31
- Neural net flies a drone in Acro mode from streamed game footage on a tiny board — yacineMTB · 2026-10-31
- llmlip-rs: Rust MLIP inference engine runs 6000-atom sims 17.7x faster than PyTorch MACE — CatAstro_Piyush · 2026-10-31