EMNLP paper: sample-then-fetch crawling yields several times more high-quality LLM pretraining pages
metehan777 · x · 2026-10-12
Malte Ostendorff's paper "Host-Aware Web Crawling for LLM Pretraining Data" (WaC-13 @ EMNLP 2026) studies how crawlers should allocate budget across and within hosts:
- Combining quality classifier scores, web-graph centrality, and Common Crawl per-host coverage, roughly half of classifier-score variation occurs between hosts; a few pages from a host usefully estimate the quality of the rest.
- Common Crawl's harmonic centrality allocation yields limited gains over random ordering and underperforms for some classifiers.
- For crawl depth, sampling a few pages then fetching more from higher-scoring hosts collects several times more high-quality pages per fetch than uniform or centrality-based allocation.
- Common Crawl's observed per-host coverage barely correlates with quality; tens of millions of high-quality pages remain uncrawled per crawl, concentrated on a small share of hosts.
More from Infra
- Open WebUI hits 154k GitHub stars as the go-to self-hosted AI interface — open-webui · 2026-10-12
- Microsoft's BitNet inference framework for 1-bit LLMs tops 40k GitHub stars — microsoft · 2026-10-12
- StarCloud launches AI hardware into orbit to build space data centers, targeting power and cooling limits — DavidLinthicum · 2026-10-12
- AI infra engineer project list: vLLM tuning, KV-cache routing, Triton kernels — snowclipsed · 2026-10-12
- AI data center gold rush feeds worker-economy startups: 1,500 lunches a day per site — asusarla · 2026-10-12
- Chip shortage leaves Yandex data centers unfixable until at least 2028 — zacharynado · 2026-10-12