AI bot data: OpenAI and Anthropic rely heavily on llms.txt; Markdown pages only 50% of crawls
marclou · x · 2026-08-13
TrustMRR analyzed over 1 million AI bot visits in the last 30 days, finding that OpenAI and Anthropic heavily use llms.txt to answer users, index pages, and train models. Markdown pages are crawled about 50% of the time; AI bots still use HTML. The /mcp page is the most crawled by ChatGPT and Claude to answer users.
More from Infra
- Tencent squeezes 770B-parameter Hunyuan Hy4 Preview into 214 GiB at ~2.38 bits per weight — coolmum18 · 2026-10-16
- TokenSpeed MLA 2 hits NVIDIA Vera Rubin: up to 3.44x faster decode vs B200 — zhyncs42 · 2026-10-16
- US vs Chinese hyperscalers: Americans fund and schmooze, Chinese demand your BoM and cap you at 7% margin — jwt0625 · 2026-10-16
- How LiteLLM routes one OpenAI-shaped API across every model provider — Mahmoud_Zalt · 2026-10-16
- OpenAI-Broadcom custom AI chip Jalapeño named a TIME Best Invention of 2026 — itsclivetime · 2026-10-16
- Developer live-runs continual pretraining of a Luganda LLM on Colab, 41% done with loss dropping fast — weskambale · 2026-10-16