Context Rot in AI-Assisted Software Development: Repurposing Documentation Consistency for AI Configuration Artifacts
Christoph Treude, Sebastian Baltes
cs.SE, cs.AI
2026-06-08
Running DOCER unmodified on AI config files finds stale code references in 23.0% of 356 repos; 64% of inspected flags are genuine rot.
Repository-level files for coding assistants, CLAUDE.md, AGENTS.md, .cursorrules and their cousins, now supply persistent project context across sessions. They name paths, API clients, scripts, and conventions. The code keeps changing. The files often do not. The model then imports a deleted module or enforces a convention the team dropped, usually without a visible error.
The authors call this divergence context rot. The phrase has mostly described degradation inside a model's input window as token count grows. Here it is extended to versioned, repository-resident artifacts that are fed to models as project context. The underlying consistency problem is old. Software engineering has spent two decades checking READMEs, comments, API docs, architecture descriptions, and install instructions against code. The claim is concrete: that toolbox is an immediate starting point for detecting rot in AI configuration.
The empirical study covers one form of rot: referential rot, where a config file points at functions, classes, constants, scripts, or paths that no longer exist. The source is Galster et al.'s GitHub corpus of AI configuration files. After dropping empty and pointer files and requiring a recorded first-commit SHA, 4,420 repositories remain eligible. A random sample of 356 repositories (seed 42) contributes every config file in those repos, 612 files, sized for 95% confidence and a 5% margin of error on the eligible set.
The detector is DOCER, a regex-plus-two-snapshot checker built for READMEs and wikis. The authors do not tune it. Candidates are extracted at HEAD, confirmed in source at the config file's first commit, then searched again at HEAD. Present-then-absent is stale; absent at both snapshots is discarded as noise. README files and AI config types are excluded from the search scope so documentation does not match documentation. The goal is not to improve DOCER. It is to see whether an unmodified documentation checker already fires on this new artifact class.
DOCER extracted 29,454 candidate elements and verified 18,048 against source at first commit. Of those, 17,818 remain at HEAD and 230 are stale, across 82 repositories, 23.0% of the sample (95% CI 18.8–27.2%). Among affected repositories the median stale count is 1 and the maximum is 20. Per-reference stale rates by file type sit between about 1.0% and 1.4%; the paper does not test those differences.
One author inspected 50 elements classified as stale: 32 (64%) were genuine referential rot, 12 false positives, 6 ambiguous. False positives mostly come from broad regexes matching English words or generic tokens. Two opposing biases are stated: the two-snapshot rule misses references introduced in later edits to the config file, which deflates the rate; false positives inflate it. The 23.0% figure is offered as a feasibility signal, not a precise prevalence.
| Metric | Number |
| Repos sampled / with stale refs | 356 / 82 (23.0%) |
| Verified refs / stale | 18,048 / 230 |
| Genuine rot in 50-item audit | 32/50 (64%) |
For teams that already keep CLAUDE.md in git, this is a checklist, not a new algorithm. The two-snapshot git grep procedure can run in CI and catch renamed functions, deleted scripts, and dropped dependencies. Treating config files as code, and reviewing them with refactors, prevents a large class of this rot at the source. The roadmap maps comment checkers, API-doc checkers, architecture traceability, and install/dependency checkers onto behavioral instructions, MCP tool descriptions, architectural claims, and runtime versions, and lists four research questions: what kinds of rot exist, which old tools transfer unmodified, which kinds change assistant behavior, and how to repair them.
This is a short paper plus a roadmap. The empirical evidence covers only referential rot. DOCER's staleness definition requires the element to exist at the config file's first commit, so later-introduced references are out of scope. The manual audit has a single annotator and no agreement statistic. The sample is public GitHub repositories that already contain AI configuration. The larger gap is behavioral: the paper does not measure how much stale config actually degrades assistant output. Prior work by Lulla et al. shows that the presence of AGENTS.md changes runtime and tokens; a causal experiment on injected stale references remains RQ3.