Top AI Crawlers Fetched robots.txt 1,150 Times vs 4 for llms.txt
dejanseo · x · 2026-08-15
A 30-day log analysis from dejan.ai reveals that crawlers from Google, Anthropic, and OpenAI fetched robots.txt 1,150 times combined, but only accessed llms.txt 4 times and the OKF bundle 23 times. This indicates extremely low adoption of the llms.txt protocol by major AI vendors, who still rely heavily on traditional robots.txt.
More from Infra
- Plivo cuts India AI agent voice call price to ₹0.19 per 30 seconds — bevenky · 2026-08-15
- Qwen 3.8 27B Hits 51 t/s on AMD Radeon AI PRO R9700, Users Share Benchmarks — pmttyji · 2026-08-15
- Skepticism regarding Mythos inference optimization claims — scaling01 · 2026-08-15
- OpenAI's GPT-5.6 Sol on Cerebras is nearly 7x faster than Fable 5 — haider1 · 2026-08-15
- Building a Disk-First Production Pipeline in ComfyUI — sugar_shak · 2026-08-15
- Testing AF_XDP on Google Cloud C4N hits nearly 6 Mpps on one core — HankYeomans · 2026-08-15