Opening a Site to AI Agents: 820 Crawler Hits, 280 MCP Connections, 3 Real Tool Calls
S_B_B_B_K · reddit · 2026-10-10
The author made their small site agent-friendly (llms.txt, Markdown pages, an MCP server, pay-per-call USDC endpoints) and logged everything from the first days:
- AI crawlers: 820 requests in 2.5 days. Meta led with 398, then Amazon 262; ClaudeBot 74, OpenAI search bot 38, Perplexity 18. For comparison, Googlebot made 43 and Bing 89 requests. About a third of Meta's hits targeted Markdown versions
- MCP connections: 280 from 80 clients, almost all directories, scanners, and "trust score" bots — an industry indexing agent tools before real agents show up
- Real agents: just 3 tool calls (one robots.txt generation, two health-check-like)
- Payments: Coinbase's pay-per-call directory already lists 32,000 endpoints, but every paid call on the site so far was the author testing
Takeaways: the "agent internet" is currently bots indexing bots; AI crawlers already read your content and prefer clean Markdown; per-call payments work technically, but agents with wallets remain rare.
More from coding & agent
- Open-source script measures whether raising Claude Code subagent prompt cache TTL from 5m to 1h pays off — daniel_mac8 · 2026-10-11
- Claude Opus 5.5 Takes #1 on Image-to-WebDev Arena, Priced 60% Below GPT-6 Astra — arena · 2026-10-11
- Celesto: open-source platform gives AI agents their own cloud computer in secure microVM sandboxes — aniketmaurya · 2026-10-11
- SpIDER paper boosts code retrieval for coding agents via semantic search plus code graphs — mangahomanga · 2026-10-11
- Shipping AI Features: The Messy Reality Behind the 'Pick a Model and Launch' Fantasy — aakashgupta · 2026-10-11
- Perplexity Computer orchestrates 3 frontier models, 210k credits, to build app from 52k earnings transcripts — AravSrinivas · 2026-10-11