Cloudflare-OpenAI pilot has surfaced hundreds of millions of new pages for ChatGPT search
Cloudflare Blog · rss · 2026-10-15
- Cloudflare and OpenAI shared early results from their July research pilot: using signals from 1% of Cloudflare's network, OpenAI has discovered and indexed hundreds of millions of previously unknown web pages, expanding what ChatGPT Search can cite.
- Cloudflare signals filter out wasted crawls (nonexistent files, catch-all redirects, duplicate content), flag potentially compromised pages (like planted fake login forms) without penalizing whole domains, and provide change signals so crawlers only revisit pages worth fetching.
- OpenAI independently evaluates and ranks discovered pages; the work is metadata-based and doesn't share page content for model training.
- Next phase: better page selection, faster change signals, and measuring downstream business outcomes for site owners. Domain owners retain control via AI Crawl Control.
More from Infra
- Cadence announces sovereign AI cloud for European chip designers — karlfreund · 2026-10-23
- SGLang Deep-Dive Blog Series Breaks Down KV Cache, Posts up to 15.5x Throughput Gains — BanghuaZ · 2026-10-23
- DynaLM: Open-Source CPU-First LLM Inference Engine Seeks Feedback — Unlikely_Safety_7456 · 2026-10-23
- Capital Is Now the AI Buildout Bottleneck, Says Analyst Citing Jensen Huang — JOBhakdi · 2026-10-23
- Cerebras explains how it deploys trillion-parameter models on wafer-scale chips — bookwormengr · 2026-10-23
- IFM open-sources RL360, the agentic RL infra behind K2 Horizon's training — rupspace · 2026-10-23