When Models Edit Too Much: On the Fidelity of Minimal Code Edits
Tongyao Zhu, Wei Hern Lim, Min-Yen Kan
EMNLP 2026 (Main)
cs.SE, cs.AI, cs.CL
2026-09-04
NUS injects known-minimal bugs into 400 BigCodeBench tasks. A preserve-original prompt drops excess Levenshtein from 0.195 to 0.131 and lifts Pass@1 by 2.3.
Coding models are ranked on Pass@1. Repository benchmarks still mostly ask whether tests pass. In maintenance work a patch also has to be reviewable: small, and unwilling to rewrite design that was not broken. A one-line off-by-one can be fixed by changing one line. GPT-5.4 deletes five and inserts sixty lines of validation, dtype coercion, and resampling. All five tests pass. None of that extra code is in the spec.
The paper names this over-editing and turns it into a measurable axis: how much extra a model rewrites relative to a known minimal patch.
Four hundred BigCodeBench problems receive one or two AST-level corruptions of a reference solution; only programs that then fail the original tests are kept. The gold patch is the reversal of those corruptions. Functions average 10.4 executable lines; 91.8% of gold patches touch at most two tokens; none spans more than two lines.
Three metrics: Pass@1; excess token-level Levenshtein distance ELev against the gold patch; added cognitive complexity. Edit metrics are computed on passing repairs only. Three developers with five to ten years of experience, on 100 blinded pairs, match ELev to the majority vote for reviewability in 94.8% of decided cases and for faithfulness in 96.9%.
Two prompts are compared: “fix the function,” versus the same request plus “keep as much of the original code as possible.” The second half of the paper trains Qwen3-4B on corrupted DeepCoder with SFT, rejection-sampled SFT, DPO, and GRPO-style RL that rewards execution plus edit minimality, then tests on 20 held-out corruption families.
Under the generic prompt, correctness and fidelity split. GPT-5.5 High reaches Pass@1 0.823 with excess distance more than four times Claude Opus 4.7. Its own excess falls from 0.299 to 0.159 once preservation is explicit. All 50 frontier settings move toward smaller edits; 40 of 50 also raise Pass@1. Mean ELev drops from 0.195 to 0.131, added cognitive complexity falls 26.6%, and Pass@1 rises 2.3 points.
Reasoning modes and scale are not a single knob. Qwen2.5-Coder from 14B to 32B raises excess distance on passing repairs from 0.108 to 0.127 under the generic prompt. Slice-bound bugs hit Pass@1 0.874 and the worst ELev, 0.353: a one-token bug looks like a missing precondition, so the model rewrites the data path. Among 530 high-excess passing patches, defensive generalization appears in 64.2% and data-flow rewrites in 63.2%.
On the training split, SFT reaches in-domain Pass@1 0.932, then 0.458 out of domain, and LiveCodeBench v6 falls from 32.6% to 17.7%. RL holds 0.782 OOD Pass@1 at ELev 0.050 and LiveCodeBench 33.2%, the only recipe that does not tax general coding. LoRA rank 64 nearly matches full-parameter RL. On single-method Defects4J the pass rate barely moves; token edits for the 4B model fall from 51.9 to 35.3, while absolute pass rate stays near 7%.
Code repair now has a second ruler. Tests alone let a model ship a “more robust” rewrite that a reviewer still has to peel apart. The cheapest product fix is a preservation clause in the prompt. To bake the habit in, RL generalizes better than SFT and does not erase broader coding skill.
The scope is local function repair. Open-ended refactors are supposed to be large; they are outside this benchmark.
Corruptions are synthetic, function-level, and mostly Python, cleaner than real multi-file defects. Training is almost entirely on Qwen. Human studies are small: three annotators on 100 pairs, and one annotator on the high-excess audit. Defects4J shows the preference transfers to Java; absolute repair rates at these sizes stay low. Gold is defined as reversing the injection. A blinded audit found truly unnecessary edits in 82.3% of determinate high-excess cases; the rest are valid alternative fixes that ELev still counts as excess.