GPS Makes RL Post-Training More Compute-Efficient

机器之心 · wechat · 2026-07-11

This work proposes GPS (GeneralizablePredictivePromptSelection), aiming to reduce expensive rollout costs during RLVR post-training while maintaining or even improving reasoning capabilities.

Core Idea

First, train a lightweight Prompt Predictive Model (PPM) to predict the difficulty of various prompts for the current model.
Then, combine the principle that "medium difficulty yields higher training value" with batch diversity to select more valuable training samples, rather than relying on random sampling.
Introduce a global difficulty context during training, allowing difficulty prediction to reuse the entire optimization history instead of just tracking individual problems.

Key Results

Achieves a 1.4×–2.0× speedup in training steps compared to random sampling.
Reduces rollout costs by up to 69% and cuts training time by 28%–47% compared to DS/DSOracle, which requires actual extra evaluations.
Matches or exceeds strong baselines in math and logical reasoning tasks while being more compute-efficient.

Additional Findings

This PPM is not limited to training; it can also allocate compute budgets during testing: assigning more Best-of sampling to problems worth computing.
Brings up to a 3.2% improvement under a fixed budget, or saves 36.4% of inference compute without performance drops.

Experimental Scope

Math reasoning: DeepScaler, MATH500, OlympiadBench, MinervaMath, AMC23, AIME24, etc.
Logical reasoning: CountdownNumberGame, CD34, CD4, etc.
Compatible with various RL post-training pipelines like GRPO, PPO, and Reinforce++.

Original post →

GPS Makes RL Post-Training More Compute-Efficient

More from Infra