Comic explains GRPO's credit assignment flaw: irrelevant tokens get the same advantage
heghbalz · x · 2026-11-09
A four-panel comic circulated in a retweet illustrates the core credit assignment problem of outcome-only GRPO:
- In the example, a model solves an equation, breaks into song ("Never Gonna Give You Up"), goes on a 2,000-token tangent about loving charts, then answers correctly with \boxed{5}.
- Since the answer is right, the whole rollout gets reward 1 and a positive advantage A.
- The flaw: GRPO applies that same positive advantage to every token in the rollout — including the song and the tangent, which have nothing to do with the answer.
This captures GRPO's key weakness under outcome-only rewards: without per-step credit assignment, irrelevant content gets reinforced alongside correct answers.
More from Research
- Brian Armstrong: publicly funded science should work like open source — every experiment a repo — alishbaimran_ · 2026-11-09
- Weaver speculative decoding hits 4.08x speedup; authors cry foul as Tencent's DSpine skips their citation — teortaxesTex · 2026-11-09
- New theory shows prover-conjecturer self-play can grow proved theorems exponentially — StanfordAILab · 2026-11-09
- AbridgeKV: latent KV cache compression matches full accuracy at 1/4 cache, 3.2× faster — DimitrisPapail · 2026-11-09
- Next-Concept Modeling paper claims parallel coarse-to-fine generation could run frontier-level AI on 32GB VRAM — Lorian0x7 · 2026-11-09
- Stanford's Hinted Scaling Laws cut LLM capability forecast error from 21% to 4.3% — burny_tech · 2026-11-09