Meta AI proposes Off-Context GRPO to fix GRPO's zero-gradient 'learning cliff' on hard problems
ceciletamura · x · 2026-11-05
Meta AI and Columbia researchers (Priyank Agrawal et al.) released a preprint introducing Off-Context GRPO (OC-GRPO).
The problem: a learning cliff. GRPO learns from within-group reward variance, so on sufficiently hard problems every rollout fails, rewards are uniformly zero, advantages are zero, and the gradient signal is exactly zero no matter how long training continues. The authors construct the sharpest version: 595 MATH Level 3-5 problems where Qwen2.5-7B-Instruct fails all 64 sampled attempts.
The method. Injecting privileged information (e.g. a reference-solution prefix) into training prompts makes correct solutions reachable, but the resulting rollouts come from a distribution deployment never sees. OC-GRPO keeps the guided rollouts and reweights their advantages with an importance ratio: successes made likely by the guidance earn less credit, and failures that persist despite it are penalized more, keeping the objective honest.
A research forum discussion with Cecile Tamura accompanies the paper.
Related event: Meta AI Proposes OC-GRPO to Tackle Zero-Reward Learning Cliff(2 posts)→
More from Research
- VA-TDM: Velocity Adversarial Distillation Sets SOTA for Few-Step Diffusion Models — Yihong Luo · 2026-11-05
- SleuthVLM beats Gemini 3.8 Flash on long-video QA without ever hearing audio — antgroup · 2026-11-05
- AnchorComplete reconstructs complete scene meshes from partial point clouds — Weirong Chen · 2026-11-05
- Neuroscientists debate using deep RL virtual animals as neuroscience research tools — neuroecology · 2026-11-05
- Berkeley team shows LLMs lose info via recursive self-summarization, fixes it with RL — berkeley_ai · 2026-11-05
- ReproEvalCard: a reporting standard for reproducible LLM pipeline evaluation — kalyan_kpl · 2026-11-05