Meta AI’s OC-GRPO adds 13.8% relative gain on Qwen2.5-7B-Instruct

bronzeagepapi · x · 2026-07-23

A Meta AI paper introduces Off-Context GRPO (OC-GRPO), a small modification to GRPO designed for hard RLVR problems where unguided rollouts get zero reward.

Core idea

Use privileged guidance during training, such as solution prefixes or hints, to obtain successful rollouts.
Keep the training objective tied to the original unguided prompt by reweighting the guided samples with an importance correction.
This addresses the “off-context” mismatch that can destabilize naive guided training.

Key findings

The paper gives a theoretical interpretation of the correction as a behavior-aware credit assignment method.
The estimator’s variance depends on the length of the guidance text, not the full rollout length.
Without this correction, guided baselines like PrefixRL and BREAD can underperform vanilla GRPO on smaller models.
On standard math reasoning benchmarks, OC-GRPO reports a 3.9% absolute gain and 13.8% relative improvement over vanilla GRPO on average for Qwen2.5-7B-Instruct across AIME, Gaokao, and OmniMath.

Takeaway

The method suggests that guided RL training only works reliably when the update remains aligned with the unguided objective.

Original post →

Meta AI’s OC-GRPO adds 13.8% relative gain on Qwen2.5-7B-Instruct

More from Research