Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models
Andrej Andrejev
cs.SE, cs.AI
2026-09-05
On a shared Flutter/Dart set, whole-file generation beats iterative search/replace for a 100M from-scratch model and Qwen2.5-Coder-0.5B, by 45.5 and 39.9 dart_pass points.
When a code model edits an existing file, it can emit the whole modified file, or it can emit a sequence of search/replace hunks, apply one, then generate the next, until it stops or hits a step budget. Production coding agents and IDE plugins mostly take the second path. A diff is shorter than a file, looks like a pull request, and should cost fewer generated tokens. Whether that intuition holds as a training objective remains less settled. Nobody had run both regimes end to end on the same models and the same data.
Independent researcher Andrej Andrejev does that for Flutter/Dart editing. Two backbones: Rainbow-Pony-100M, trained from scratch, and a fine-tuned Qwen2.5-Coder-0.5B. Each backbone is trained in a direct arm and a steps arm, four models total, sharing the task pool, tokenizer pipeline, and evaluation harness.
The source pool is 14,600 hand-written Flutter tasks: a goal, an initial file, and a final file, across 36 types and three complexity tiers. Direct fine-tuning trains on those full files, about 5 million tokens over 5,000 steps. Steps decomposes the same examples into search/replace trajectories, in the spirit of LintSeq, and trains on about 50 million tokens over 43,000 steps, roughly 10× the token budget. The step cap is 20. Application is exact match. If a search span hits more than once, a first-occurrence heuristic disambiguates. The author flags this as a patch that can silently edit the wrong site.
Evaluation is about 1,790 held-out tasks under greedy decoding. The primary metric is whether dart analyze passes. They also log bits-per-byte, character similarity, and, for steps, why the trajectory ended. gpt-4.1 scores goal fulfillment, correctness, and code quality on a 1–5 scale, blinded to model name, mode, and compile outcome, 7,161 judged rows in total.
Whole-file generation wins on every metric.
| Model | Mode | dartpass | bits/byte | similarity |
| Rainbow-Pony-100M | direct | 80.2% | 0.107 | 0.511 |
| Rainbow-Pony-100M | steps | 34.7% | 0.180 | 0.439 |
| Qwen2.5-Coder-0.5B | direct | 90.0% | 0.088 | 0.568 |
| Qwen2.5-Coder-0.5B | steps | 50.1% | 0.142 | 0.493 |
The dartpass gaps are 45.5 and 39.9 points, with non-overlapping confidence intervals. 81–85% of steps trajectories finish with stopreason=done, so this is not mostly budget exhaustion or parse failure. About 84% of Rainbow-Pony steps failures and 70% of Qwen steps failures sit inside those completed trajectories: the file was rewritten badly in silence.
The fallback heuristic is a large share of that. Among done trajectories, dartpass is 57.0% / 80.0% with no fallback and 12.3% / 17.5% once a fallback fires.
A matched-ID slice of clean steps runs (done, no fallback, under 20 steps) versus direct on the same IDs makes the gap larger, not smaller: 57.2% versus 82.1% for Rainbow-Pony (24.9 points), 80.0% versus 92.4% for Qwen (12.4). Restrict further to rows that compile on both sides, and the blinded judge still prefers direct on all six comparisons, p<0.001. Goal fulfillment is 3.44 versus 3.89 for Rainbow-Pony and 4.36 versus 4.72 for Qwen.
Steps is not uniformly worse. It wins on dartpass for only 3.5–4.8% of rows, and those wins sit on short trajectories: mean 5.54 / 6.65 steps when it takes a majority of judge dimensions, versus 8.94 / 9.02 otherwise. Refactoring and error-handling/edge-case fixes are the only two categories over-represented in both architectures, and they are the two lowest mean-step categories of nine (refactoring 4.14 / 4.49). The author calls the shared axis task locality: how spatially narrow and self-contained the required change is.
For small models trained to edit code without compiler feedback between steps, whole-file generation is the better default. Token-saving arguments from inference-time tools such as Aider do not automatically transfer to the training objective. LintSeq found edit-sequence training helpful for synthesizing a program from a blank slate. Editing an already-working file is a different job: one bad disambiguation corrupts code that used to be correct.
The operational takeaway is to pick a format per edit, not per model. Short, local refactors and edge-case patches can stay on diffs. Multi-site changes should go whole-file. That lines up with Cheng et al. 2026 on adaptive output formats.
Steps still loses after a 10× token budget, which cuts against a simple undertraining story, but longer trajectories may still be under-sampled inside the steps set, so the locality result wants a token-matched rerun. Results are Flutter/Dart only, greedy, one sample, and steps never sees compiler or test feedback between edits, so they should not be read as a claim about SWE-bench-style tool-using repair loops. A stricter apply harness that rejects ambiguous searches might shrink the gap. The judge has no human-agreement check. qwen-direct was trained under a mis-specified cosine schedule that barely decayed; it is still the best arm, and the paper reports the checkpoint it actually produced.