LLMs Corrupt Your Documents When You Delegate
Philippe Laban, Tobias Schnabel, Jennifer Neville
cs.CL, cs.HC
2026-04-17
DELEGATE-52 simulates long delegated editing via reversible round-trips across 52 domains. Across 19 LLMs, frontier models corrupt 25% of document content after 20 rounds, and agentic tools make it worse.
Vibe coding has turned "hand the whole job to the model" into a mainstream interaction pattern. Delegation rests on trust: you expect the model to edit a document faithfully without introducing errors. But over a long workflow the model degrades the document a little at a time, and you may lack the time or the expertise to check every change. This paper asks whether current models are reliable delegates at all.
Prior LLM benchmarks mostly look at single-turn QA or one-shot coding tasks. There was no benchmark that repeatedly edits the same document, spans many professional domains, and scores objectively. The authors see that gap and fill it.
DELEGATE-52 spans 52 professional domains (coding, crystallography, genealogy, music notation, accounting, recipes, and so on) with 310 work environments. Each environment has a 2–5k-token seed document, 5–10 complex edit tasks, and 8–12k tokens of distractor files.
The core device is the reversible round-trip. Every edit task comes with a forward instruction and its inverse: apply forward s→σ(s), then inverse, and perfect execution should recover the original. Chaining 10 round-trips into a relay gives 20 interactions. Scoring is then objective by construction; higher reconstruction is better, and the main metric is RS@k, the reconstruction score after k interactions. Each domain ships its own parser and similarity function; the recipe domain weights ingredients 40%, steps 40%, and tips 20%.
Reversibility is deliberate. It makes "faithful execution" objectively measurable, at the cost of excluding subjective or many-to-many edits.
19 models from six families were tested. Reconstruction scores for frontier models after the 20-round relay:
| Model | RS@20 (reconstruction) |
| Gemini 3.1 Pro | 80.9% |
| Claude 4.6 Opus | 73.1% |
| GPT 5.4 | 71.5% |
| GPT 5 Nano | 10.0% |
The "ready" threshold the authors set is RS ≥ 98%. Most models clear it only in Python (17 of 19); Gemini 3.1 Pro is ready in 11 of 52 domains. Across all models the average degradation is about 50%.
Three factors push scores down. Document size: GPT 5.4 drops from 91.4% at 1k tokens to 59.9% at 10k; each extra 1k tokens costs roughly 0.7% after 2 interactions but 3.6% after 20. Interaction length: stretched to 100 interactions, GPT 5.4 falls to 58.7%, GPT 5.2 to 50.4%, GPT 4.1 to 33.3%, with no model plateauing. Distractors: removing them lifts GPT 5.4 from 71.5% to 77.8%.
Giving models agentic tools (file read/write, code execution) does not help. It hurts, adding about 6 points of degradation on average (GPT 5.4: 71.5%→68.3%; GPT 4.1: 49.5%→40.4%) while burning 2–5× more input tokens.
These numbers puncture the assumption that models can already take over long tasks reliably. Handing a document to the model and reading only the final version carries structural risk: errors are sparse but severe, and they compound over long interactions. For anyone building agents the takeaway is concrete. Long workflows need checkpoints and rollback; do not assume the model stays self-consistent. And bolting tools onto an agent does not make it more reliable; sometimes it just packages the same errors at higher cost.
The authors list several. Each step is an independent single-turn session, so multi-turn memory is not modeled. Only textual, encodable, reversible documents are covered; subjective or many-to-many edits are out. The round-trip relay scores edit-shaped tasks kindly and may be unfair to open-ended creation.
One more concern. The 25% figure is the mean of three frontier models, a small sample. Variation across domains is large (coding is near-perfect, music notation and crystallography are poor), so the average hides the spread. "Corruption" is also computed with domain-specific similarity functions, so cross-domain comparability is limited.