Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination
Ruijie Hou, Yueyang Jiao, Zhao Wang, Yingming Li
cs.CL, cs.AI
2026-08-07
The standard G-AP metric overstates contamination fixes ~14×. SA-PPG samples per-question solve probability and stratifies the gap; RailCap, suppressing memorization at decode, ranks first.
Test questions from public benchmarks leak into the pretraining corpora of large models; the model memorizes them and scores higher than it should. A line of work does contamination mitigation, intervening at decoding time to suppress memorization and recover the model's genuine ability. The open question is how to measure whether the mitigation actually worked.
The standard metric is G-AP (gap of aggregate performance): read the model's average score on the full test set before and after mitigation, then subtract. The authors show G-AP has two hard flaws. First, it represents each question with a discrete correct-or-incorrect mark, but repeated samples on the same question disagree, so the 0/1 readout has large variance; in a reproducibility check the mean per-question gap between two batches reaches 0.190 under discrete readout but only 0.041 under probability readout. Second, G-AP averages over the full set before differencing, so under-suppression on one question cancels over-suppression on another, and a zero average gap does not mean every question is restored. Existing strategies may look good only because the metric hides the problem.
Two contributions: a harder-to-game metric (SA-PPG) and a mitigation strategy (RailCap).
SA-PPG (stratified aggregate of per-question probability gaps). For each question, sample m times (m=50 in the experiments) and estimate the model's solve probability by the fraction correct; do this for both the contaminated and a clean model, difference per question to get a per-question probability gap, then partition questions into 50 equal-width bins by the clean model's solve probability and average the absolute gap within each bin before averaging across bins. The stratification closes a loophole: a strategy could push solve probabilities only on questions the clean model already solves (high-frequency values), and an unstratified average would be fooled by mass concentrated there.
RailCap (suppress contamination during generation). Two observations drive it. On leaked questions, the contaminated model's sampled responses collapse onto its own greedy trajectory (verbatim recall), and at divergence steps the clean model's token is, about half the time, the contaminated model's runner-up. RailCap therefore turns contamination detection from a one-shot pre-hoc estimate into step-wise supervision: build an n-gram index from the greedy trajectory, and at each decoding step check whether the last n tokens match a window in that index; if they do, cap the next trajectory token's logit to the second-largest, accumulating suppression until the response distribution is dispersed. No need to guess in advance which questions are contaminated.
Models: Llama-2-7B, Gemma-4-E2B, Pythia-12B. The clean model is fine-tuned on 1,840 OpenOrca plus 660 GSM8K training questions; the contaminated model is additionally trained on 660 leaked test questions. Evaluation uses GSM8K test (1,319 questions) and a paraphrased version (PQ).
The headline is the rank reversal. On Llama-2 × GSM8K:
| Strategy | G-AP | SA-PPG |
| Identity | 0.3192 | 0.3261 |
| LNE-blocking | 0.0235 | 0.2932 |
| Shortcut | 0.1812 | 0.2476 |
| RailCap | 0.0728 | 0.1914 |
LNE-blocking looks near-perfect under G-AP (0.0235) but falls to 0.2932 under SA-PPG, barely better than no intervention, dropping behind RailCap and Shortcut. RailCap is not the best under G-AP (0.0728) but is the best under SA-PPG (0.1914). Same questions, same model, and the ranking flips when the metric changes. Across all six settings (3 models × 2 benchmarks) RailCap has the lowest SA-PPG. LNE-blocking's restoration is overestimated by a factor of about 14 (0.0207 under G-APP versus 0.2932 under SA-PPG).
For anyone running benchmark evaluations, the central warning is that your contamination mitigation may be less clean than you think, because the old metric lets over- and under-suppression cancel out. SA-PPG is a cheap diagnostic (a few dozen samples per question) worth running after any mitigation. RailCap's idea also stands on its own: rather than pre-judging which questions are dirty, it applies pressure at decode time based on whether the sample has collapsed onto the greedy trajectory, replacing a static contamination estimate with real-time supervision.
This is a measurement-and-single-strategy improvement, not a complete fix. RailCap is best in every setting, yet its SA-PPG still sits at 0.16 to 0.23, well short of true restoration, which says the ceiling of current mitigation methods themselves is low.
The paper has no explicit limitations section. From the content, the costs and scope bear questioning. SA-PPG needs 50 samples per question plus one greedy decode and a same-domain clean model as reference, a real compute cost for large models; the experiments cover only math reasoning (GSM8K and its paraphrase), with no validation on code, long-form, or multiple-choice; RailCap's n-gram window n needs tuning (the paper reports n from 3 to 7 all within 0.008); and PQ paraphrase quality depends on DeepSeek-V4-Flash, which can inject noise. The clean model is a same-architecture, same-data reference; for a black-box production model, such a reference may not be obtainable.