Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency
Parsa Mazaheri, Kasra Mazaheri
cs.AI, cs.CL
2026-08-17
One audit-repair episode in context cuts an LLM verifier's false alarms in 15/15 runs by 2.8-11.5 pp; the shift sits in the response criterion, not discrimination.
Automated pipelines increasingly wire one LLM as the checker and another (or the same one) as the fixer, often inside one session: audit a problem, repair it, move on to the next audit. The paper asks whether that wiring changes what the checker reports. Prior work on accumulated messages predicts that judgments drift toward the polarity of preceding context, with negative history pulling 1.52x harder, so an audit that reported an error should make the model flag more on the next item. The measurement goes the other way.
The measurement surface is clean: ProcessBench traces whose every step is labeled correct. Any reported error is by construction a false alarm, so no fresh gold labels are needed.
Before the target audit task, the context carries a two-turn episode in which the model itself audits a problem and produces a repaired solution. The control is not an empty context but a length-matched non-audit exchange (restate the problem, do not solve it), which separates "more context" from "audit experience". Three open-weight models (Qwen3.6-27B, Qwen3.6-35B-A3B, Ministral-3-14B) cross five semantically matched wordings of the audit instruction: 15 combinations. False-alarm rate uses 8 samples per item at T=0.7; at T=0 each item collapses to a hard 0/1 and small shifts in reporting propensity disappear into that boundary.
Two decompositions follow. Signal-detection analysis measures false alarms on correct traces alongside detection on labelled-incorrect ones, splitting any change into criterion c (the flag/no-flag threshold) and d-prime (discrimination). Component tests separate the audit verdict text, the repair content, and the repair request; which component carries the effect differs across model families.
All 15 combinations move the same way: a prior audit-repair episode lowers false alarms by 2.8 to 11.5 percentage points, a 9 to 25% relative drop. At the primary wording, Ministral-3-14B falls 8.83 pp; the two Qwens fall 4.00 and 3.59 pp.
The shift sits in the threshold, not discrimination. Criterion c moves toward leniency in 15 of 15 combinations and survives its statistical correction in 13, while delta d-prime survives in 0 of 15. The authors add the caveat themselves: the d-prime test carries twice the standard error of the criterion test by construction, so "discrimination unchanged" is unproven rather than established.
The counterintuitive wedge: an episode whose audit reported an error lowers false alarms further still. On Ministral, error-verdict episode minus clean-verdict episode is -5.62 pp, negative at all five wordings. The negativity-asymmetry literature predicts the opposite sign.
A hand audit of 50 false alarms finds 41 (82%) simply wrong, 8 (16%) defensible-but-stricter readings, and 1 suspected gold-label error. At this operating point, leniency mostly cleans up noise. With reasoning enabled the relative effect holds on both Qwens: -19.7% and -17.5%, against -21.6% and -14.3% with reasoning off.
An LLM verifier's verdict depends on its context history as much as on the task in front of it. Pipelines that audit many items in one session, from agent self-correction loops to automated code review and batch grading, drift systematically toward leniency, and in a direction the accumulated-message literature gets backwards. Treat what came before in the context as a first-class control, on par with wording and temperature. The measurement design here (length-matched filler, five wordings, signal-detection split) is a reusable template.
From the authors: the error-verdict wedge holds on only one of three models; the reasoning arm covers two; the 50-item hand audit is single-author with no agreement check; two screened-out Gemma candidates moved the opposite way, so the direction is not stable across families; all models are open-weight, with no frontier or closed judges tested.
Reading it critically: one domain only (math process verification), with no extrapolation to code review or fact-checking; and whether a 9 to 25% relative drop matters depends on how much a missed error costs relative to a false alarm, a trade-off the paper does not analyze.