Meta Paper Reveals Why RL Fails at Code Optimization and How to Fix It
rohanpaul_ai · x · 2026-08-01
A new paper from Meta explains why applying reinforcement learning directly to code optimization usually fails: runtime is a sparse and noisy signal, and naive reward mechanisms often degrade code correctness.
The authors argue that the entire feedback path must be co-designed:
- Infrastructure: Scale up optimization tests and use a calibrated remote execution service for accurate timing.
- Rewards: Implement problem-relative ranking against human solutions with correctness-gated binary rewards.
- GRPO Updates: Stabilize training for noisy zero-advantage batches by increasing samples per prompt, enlarging batch sizes, dropping group variance normalization, and discarding stale timing data.
After rebuilding this full pipeline, the Qwen 2.5 7B model improved its success rate at the top-50% speed threshold significantly, jumping from 18.0% to 31.3%.
More from Research
- It's not gradient descent: burny_tech corrects the diagnosis to RLHF mode collapse — burny_tech · 2026-10-16
- A beginner-friendly introduction to causal inference kicks off, Part 1 — Nasereliver · 2026-10-16
- FBI method trains for Pass@K with one sample per update at 12.3% of the cost — EmtiyazKhan · 2026-10-16
- Berkeley researchers sample humanoid robot policies directly from weight space with diffusion — berkeley_ai · 2026-10-16
- OneWM: unified causal training shows world and action modeling improve each other — IAAI233 · 2026-10-16
- Baidu's Virtual Head Attention tames the KV cache, FLOPs, quality triangle in long-context LLMs — baidu · 2026-10-16