ExTra adds novelty rewards and entropy-guided regeneration to GRPO, +5 pass@1 on Qwen3-1.7B
burny_tech · x · 2026-11-10
Researchers from the National University of Singapore introduce ExTra (Exploratory Trajectory Optimization), a GRPO-compatible exploration framework addressing RLVR's failure at both task-difficulty extremes: easy prompts yield all-correct, low-diversity rollout groups with little gradient signal, while hard prompts produce all-incorrect groups with no positive reward.
Two mechanisms:
- Novelty reward: embedding-based diversity bonuses added after GRPO normalization, rewarding diverse correct solutions (for easy problems);
- Entropy-guided prefix regeneration: scores partial trajectories via mean token entropy and resumes exploration from promising low-entropy intermediate steps (for hard problems).
Results: across six math reasoning benchmarks, ExTra lifts Qwen3-1.7B over GRPO by +5 points on pass@1 and +7 on pass@16, showing trajectory-level exploration signals improve both single-sample accuracy and inference-time coverage.
More from Research
- Quanta: Injecting 'thoughts' into LLMs to test whether AI can introspect — burny_tech · 2026-11-10
- Two CoRL papers speed up VLAs 2.4x and 4.7x, fast enough for real-time robot control — songhan_mit · 2026-11-10
- Four Free Math Books for ML: Linear Algebra, Probability, Optimization, Information Theory — cneuralnetwork · 2026-11-10
- 4 free math books that actually matter for ML: linear algebra to information theory — cneuralnetwork · 2026-11-10
- Meta FAIR's e-means beats k-means in 1/10 epochs with 10-45x GPU speedups — burny_tech · 2026-11-10
- NeurIPS workshop paper: gradient-based data attribution can't reliably filter subliminal learning — ponguru · 2026-11-10