When Search Eats the Web: A Model of Corpus Erosion under Generative Extraction
Sylvain Peyronnet
cs.GT, cs.IR
2026-08-17
A game-theory model treats the crawlable web as a renewable common-pool resource and proves a single erosion threshold separates survival from extinction, with competing engines pushing equilibrium extraction toward the threshold and dissipating all rent.
Search engines and publishers have an implicit contract: crawl, index, rank, then send the user to the source, which monetizes the visit. That contract financed the web's content for two decades. Generative search engines (GSEs) break it by synthesizing a direct answer from crawled content and keeping the user in their own interface. Existing work measures who loses what; one study puts the daily traffic loss for Wikipedia articles exposed to Google AI Overviews at roughly 15%.
This paper asks a different question: not who loses what today, but what happens to the crawlable corpus itself. Which content stops being crawlable, what stops being produced, and how the lifetime of what remains changes. Sylvain Peyronnet (independent lab IBOU) builds it as a formal economic model.
The central abstraction treats the crawlable corpus as a common-pool resource, the category fisheries and forests belong to, where anyone can extract and nobody pays for maintenance. It is described by volume N, average quality Q, and lifetime (a depreciation rate lambda). Publishers are distributed by quality theta; the engine has an extraction rate e, the fraction of a page's value the direct answer keeps. e=0 is classical search, e=1 is a pure direct answer.
Extraction erodes the corpus through three simultaneous channels:
All three channels feed one logistic renewal equation, which is why a single threshold e emerges: renewal equals depreciation. Below it the corpus settles into a smaller but stable steady state; at it, volume decays hyperbolically; above it, exponentially.
| Result | Content |
| Threshold | e is unique; the three channels share one critical point, not three |
| Myopic engine | The short-run optimal rate never sits below the opt-out threshold; in the parameter region e < eout, an engine maximizing answer quality at every instant drives its long-run payoff to zero |
| Competition | With m engines, the symmetric equilibrium rate is nondecreasing in m and converges to e; aggregate payoff goes to zero as m grows, i.e. full rent dissipation |
| Social optimum | Even under the assumption most favorable to extraction (users strictly prefer direct answers), the socially optimal rate lies strictly below e and no higher than the single engine's sustainable optimum |
| Full ordering | eusers ≤ esoc ≤ eeng = ê(1) ≤ ê(m) < e, with the myopic rate beyond the threshold |
In the paper's explicit parameter instance, the single-engine sustainable optimum is about 0.537, five engines sit near 0.690, fifty near 0.724, against a threshold near 0.738. Competition needs no myopia at all to push the system to the cliff edge: each engine bears the full cost of its own restraint but collects only 1/m of the value that restraint preserves. The tragedy of the commons, restated for the crawlable corpus.
The paper replaces a debate about whether AI search will kill the web with a parameterized model. Its three thresholds (e, eout, e) map onto levers institutions can actually move: engines can lower e unilaterally (referral floors, teaser answers), market mechanisms can raise renewal (flow-indexed licensing, Shapley-based attribution payments), and regulation can cut coordination costs (the EU press publishers' right already exists). Seven survival mechanisms are laid out, grouped by how much coordination each requires.
For engineers building RAG or generative search products, the direct lesson is that the corpus is not a fixed input but a function of your extraction rate. The answer-quality optimum and the corpus-survival optimum do not coincide, and the gap widens with every competitor that enters.
The author's own caveats: queries are split equally across engines, whereas market-share concentration is what matters in reality; the average-payoff criterion ignores the transition, during which engines still profit from the declining stock; users are deliberately myopic, so the model cannot weigh the convenience gain of direct answers against erosion.
Beyond those, the model is entirely symbolic: not one parameter is calibrated on real data. The appendix's numerical instance was chosen to prove a concavity assumption is satisfiable, not to measure anything. The decay channel, the heaviest of the three, rests on a single linear assumption with no empirical support. Assumption 4 (concavity) is acknowledged not to follow from the rest, and the competition theorem depends on it. Depreciation also applies uniformly to old and new content, which overstates how fast the archive dies. This reads closer to a framework paper that gives the debate a skeleton than a tested prediction.