AI bot data: OpenAI and Anthropic rely heavily on llms.txt; Markdown pages only 50% of crawls
marclou · x · 2026-08-13
TrustMRR analyzed over 1 million AI bot visits in the last 30 days, finding that OpenAI and Anthropic heavily use llms.txt to answer users, index pages, and train models. Markdown pages are crawled about 50% of the time; AI bots still use HTML. The /mcp page is the most crawled by ChatGPT and Claude to answer users.
More from Infra
- LifeOS: A Local, Voice-Driven Personal Organizer — Extension-Bid-639 · 2026-08-24
- Hyperscalers: Choosing Between HDD and SSD Based on Space and Cost — generativist · 2026-08-24
- Samsung shows new HBM cooling solution, hints at die performance variance — BenBajarin · 2026-08-24
- Tobi open-sources walgit: A single-binary Git server backed by object stores — jevon · 2026-08-24
- s3collections: Durable Go data structures backed directly by S3-compatible storage — andersonbcdefg · 2026-08-24
- Prediction market gives 68% chance of a state data center moratorium by year-end — Polymarket · 2026-08-24