Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinking
Chia-Hsuan Lee, Sihui Dai, Mingyang Zhou, Isha Slavin, Hsuan Su, Shi-Xiong Zhang, Sambit Sahu, William Campbell
cs.CL
2026-07-01
DASH adds segment-level credit to GRPO, no process labels: reward the prefix that gets it right, penalize the drift after it. 59.45% on competition math vs DR-GRPO's 58.13%.
Reasoning models (the o1 family) overthink. They hedge, abandon approaches mid-stream, and contradict themselves, burning tokens without improving the answer. The standard fix is to penalize length. Before proposing anything, the authors ran a measurement and showed the disease is not length: bucket traces by length, and within the same bucket, incorrect traces still show more numerical recomputation and self-contradiction than correct ones. Length is the symptom; flip-flopping is the disease.
They name the specific failure "answer drift": a trace reaches the correct answer partway through, then reflects its way to a wrong final answer. These traces are exactly where reflection hurts.
Fixing this the traditional way means scoring every reflection step, which needs either a trained process reward model or human annotation, both expensive. The authors' opening is that the answer checkpoints a model writes into its own trace (\boxed{}, "the answer is X") are free step labels, and comparing each to ground truth tells you whether a segment moved toward or away from correct.
DASH (Drift-Aware Advantage SHaping) is a module that drops into GRPO. In one line: slice each trace at answer checkpoints, give positive advantage to segments that end at a correct answer, give negative advantage to segments that end at an incorrect one, and let the penalty escalate the longer a negative segment runs.
GRPO normally hands an entire trace one scalar advantage (a binary right/wrong reward), broadcast to every token. DASH rewrites that broadcast. It regex-extracts a sequence of intermediate answer commitments from the trace as checkpoints, compares each to ground truth, and every segment picks up a free "led to correct/incorrect" label:
Segment advantage is reassigned by type. Positive segments get positive advantage, but credit decays geometrically across consecutive re-confirmations of the same correct answer, so the model does not learn to write the right answer five times. Negative segments get negative advantage that ramps up toward a cap, encoding "the further you wander after a correct answer, the worse." The neutral segment is handled conditionally by trace outcome: in a drift trace the prefix actually found the right answer, so it should not be punished with the suffix that failed to stop.
Drift traces also get a round of reward shaping: partial credit that ranks them above pure-incorrect and below fully correct. The whole method touches only per-token advantage, so it needs no process reward model, no learned components, no human labels, and no per-model tuning. On DR-GRPO it changes two hyperparameters (DR-GRPO's token-mean loss already penalizes length, so the explicit length penalty is switched off and the positive scale drops from 1.0 to 0.5).
Accuracy across four competition math benchmarks (base Nemotron-4B):
| Method | OlympiadBench | MinervaMath | AIME24 | AIME25 | Average |
| Nemotron-4B (off-the-shelf) | 60.9 | 55.1 | 60.5 | 46.1 | 55.65 |
| GRPO | 67.3 | 53.3 | 61.8 | 45.4 | 56.95 |
| DR-GRPO | 66.7 | 53.3 | 62.2 | 50.3 | 58.13 |
| DR-GRPO + DASH | 67.6 | 57.4 | 65.3 | 47.5 | 59.45 |
The gain lands where drift is worst. AIME25 has the highest correct-then-wrong drift rate of the four, and vanilla GRPO actually scores below the off-the-shelf base there (45.4 vs 46.1): its uniform negative advantage on drift traces punishes the valid prefix that found the answer. Adding DASH recovers it for the GRPO base (47.2). The standout is the hardest set, AIME24, where DR-GRPO+DASH reaches 65.3, up 3.1 over DR-GRPO.
Generalization: GRPO+DASH beats vanilla GRPO on all three backbones, +0.5 on Nemotron-4B, +1.5 on OLMO-3-think, +4.1 on Phi-4-reasoning-plus.
The overthinking signal profile (vs DR-GRPO): abandonment down 41% (1.07 vs 1.80), hedging density down 14%, length outliers down 17%. One signal goes the other way: self-contradiction rises 13%, but that is productive. Contradiction-then-resolution roughly doubles (0.92 vs 0.47 per trace); the model uses contradiction to diagnose errors instead of spiraling. Response length is essentially unchanged (+0.03% vs DR-GRPO), so DASH is not shortening answers, it is improving them.
Ablations: reward shaping is the most important component (removing it drops 2.9 points), length penalty is next (down 2.2), conditional-neutral handling and diminishing-returns each contribute 0.7 to 0.8.
For anyone training reasoning models with GRPO-style RL: a drop-in module that targets one common, specific failure (drift) without labels, judge models, or a new optimizer. The only requirement is that the domain exposes extractable intermediate answers. Math and code with test cases qualify; open-ended writing or planning without verifiable checkpoints do not.
The average gain is modest (about 1.3 points), concentrated on the hardest, most drift-prone problems. Treat it as a targeted scalpel, not a universal accuracy booster. Even apart from DASH itself, the answer-drift framing plus the six linguistic overthinking signals are reusable diagnostics for telling whether your own model is overthinking or mis-thinking.
The main experiments are almost all on a 4B model. The authors argue drift patterns look scale-invariant, but they do not verify that training dynamics hold at larger scale.
It depends hard on extractable intermediate answers, which restricts it to domains with verifiable checkpoints (math, code). Open-ended reasoning needs a different drift signal.
There is a real trade-off: on easy benchmarks like MATH-500 it loses 1.7 points against the base, buying hard-problem gains at the cost of easy-problem accuracy.
Only mathematical reasoning is validated; generalization to logical, scientific, or commonsense reasoning is untested.
Drift detection relies on regex extraction of answer commitments. Messy traces that never produce a clean \boxed{} or "the answer is" may not segment correctly, and the paper does not stress-test this.
One thing the paper leaves open: whether the +13% contradiction always converts into recovery on the hardest problems, or sometimes just becomes a different way to fail.