Logs show OpenAI/Anthropic bots not crawling llms.txt yet
gaganghotra_ · x · 2026-08-18
A developer's log analysis reveals that while OpenAI and Anthropic bots make up to 300K daily requests to site URLs, there are zero requests for /llms.txt.
This suggests that despite llms.txt being proposed as a standard for LLMs (analogous to robots.txt for search engines), major model providers have not adopted it into their crawling workflows yet. If they aren't actively seeking it, the standard likely has little impact on their current operations.
More from Infra
- AI Chip Startup Etched Signs Quant Giant Jane Street as First Customer — KateClarkTweets · 2026-08-18
- Agent Governance Shifts to Device Level with mimOE Engine Release — shashib · 2026-08-18
- Ex-Tesla SVP Drew Baglino breaks down how a data center burns a gigawatt of power — wandb · 2026-08-18
- Tesla Alum Raises $140M to Fix AI Power Bottleneck with Grid Engineering — wandb · 2026-08-18
- CoreWeave: Prior-Gen GPUs Sold Out, Signs A100 Contract Through 2029 — Beth_Kindig · 2026-08-18
- Minos Genomics AI Cuts Egress Costs with Hippius S3 Storage — const_reborn · 2026-08-18