When Search Eats the Web: Paper Models How Generative Extraction Erodes the Corpus

IgorCarron · x · 2026-08-18

A new arXiv paper models the impact of generative search engines (GSEs) with game theory:

Core problem: GSEs answer queries directly from crawled content without sending traffic back, defunding content creation; publishers respond by blocking crawlers.
The "crawlable commons": the corpus is modeled as a common-pool resource described by volume, average quality, and lifetime. Extraction degrades all three — publishers opt out, renewal loses funding, content perishes faster — and beyond an erosion threshold the corpus goes extinct.
A myopic GSE crosses the threshold; a long-run oriented one stays below it.
With competing engines, the symmetric equilibrium extraction rate rises with their number and converges to the threshold; even with users who prefer direct answers, the socially optimal rate stays strictly below it.
Seven survival mechanisms are discussed.

More from Safety