How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
Jenna Russell, Ben Glickenhaus, Katherine Thai, John Wieting, Mohit Iyyer, Max Spero, Bradley Emi
cs.CL, cs.LG
2026-10-01
800 pretrained models show wild AI web text helps only data-starved runs and hurts past 20 tokens/param; a new scaling law cuts prediction error 41% below the best existing one.
The bulk of pretraining data is web text, and the AI-written share is climbing fast. After FineWeb quality filtering, Pangram labels 10.1% of tokens AI-generated in June 2024, 27.5% in June 2026, and 31.1% by August 2026. Every new crawl forces a choice: keep those documents, filter them, or search for an optimal mix.
No prior work answers this directly. Model-collapse research trains a model recursively on its own output; synthetic-data research adds curated rephrasings designed to help. This paper measures a third category it calls wild AI text: written by many different models for human readers, unlabeled, mixed into crawls. That is the setting pretraining pipelines actually face.
The design goal is explicit: the value of an AI token, in human-token equivalents, should be allowed to go negative once the human budget is sufficient. Repetition laws never let token value fall below zero, and mixture laws fit harm without separating it from benefit.
| Setting | Metric | Result |
| Predicting AI additions on C4 | paired RMSE (×10⁻³) | ours 0.83; Shukor joint 1.41; Chinchilla 4.32 |
| Repeating human text vs adding AI at 20 TPPh | C4 loss gap | 2.3% lower after 1 added epoch, 6.3-6.4% after 8 |
| Mixed validation set at 22.3% AI share | harmful runs reported as improvements | 95.5% (sign flips at 5.1% share) |
| Bias of quality filters | AI docs kept vs human docs | FineWeb 2.3x, DCLM 9.8x |
| Cost of not filtering at 31.1% AI | compute-equivalent gain | 1.6x, rising to 3.0x at the 2028 forecast share |
AI text pays off only when human data is scarce: the loss-minimizing AI share on C4 is 37% at 5 tokens per parameter and 1.0% at 20. In the filtering experiments, past roughly 10-15 tokens per parameter the smaller human-only models reach lower loss, and the new law calls the direction right in 18 of 19 pairs while Chinchilla gets all 13 winnable pairs wrong. Training on AI text also changes how a model writes; at 268M, AI-typical phrases per 1,000 words climb from 0.16 to 0.54 at r=4, and at r=2 Pangram labels twice as many generated stories AI (37.2% vs 18.6%). When the target is AI text itself (Cosmopedia), the conclusion inverts: the optimal mix stays above 90% AI, and the first AI token is worth more than 20 human ones.
For anyone building pretraining data pipelines, the recommendations are directly actionable: filter AI web text when the target is human text, since modern models train far past 20 tokens per parameter and sit squarely in the harm regime; repeat human data instead of padding with AI text; report validation loss on human and AI text separately, because a mixed set flips the sign of harm above a 5.1% AI share. Current quality filters select in the wrong direction, and the mismatch grows as the AI share rises.
Author-stated: the law is fit up to 268M and validated only to 973M; larger models memorize more and may be hurt worse by model-generated data, so extrapolation to frontier sizes is untested. Only next-token loss is measured, and the models are too small for meaningful downstream benchmarks. Coverage is English web text under Pangram labels. The 8B compute comparison is a law projection; no 8B model was trained.
Two further concerns. The human/AI split leans entirely on detector accuracy, tolerable at a 0.05% false-positive rate against a 31% share, but quality differences within AI text (strong-model output vs SEO spam) are flattened. And the 50.7% share forecast for 2028 comes from a random walk with drift whose 95% interval spans 31-70%, so the 3.0x figure inherits wide uncertainty.