Optimizing Websites for AI Assistants and Agents
marclou · x · 2026-07-14
Indie developer Marc Lou shared practical tips on optimizing websites for AI assistants (like ChatGPT and Claude) and autonomous agents.
- Background: To help AI assistants better browse his startup marketplace TrustMRR, he added .md plain text pages, an llms.txt file, and a restricted /api/ai endpoint to provide structured data.
- Issue: After the initial production deployment, ChatGPT encountered cache misses or 400 timeout errors when crawling the pages.
- Solution: He added 8,831 URLs to sitemap.xml and configured Content-Type: text/plain and Content-Disposition: inline response headers. Within 24 hours, ChatGPT was crawling smoothly.
- Data: Currently, daily requests from ChatGPT alone nearly match the volume of human traffic. Developers must optimize their site structures early to prepare for the incoming wave of AI traffic.
More from Apps
- A market map tracks outpatient healthcare agentic AI across front, back and mid office — HealthcareAIGuy · 2026-07-21
- Synthesia launches Dubbing 2.0 with 130+ languages and lip-sync video translation — synthesiaIO · 2026-07-21
- User plans dozens of voice interviews with ChatGPT to build a book about themselves — mikesimmi · 2026-07-21
- Linear Launches Loops: Automate Workflows with Plain English Instructions — xiaohu · 2026-07-21
- Halliday opens priority access to G2 display AI glasses for meetings and daily use — SucceededMind · 2026-07-21
- A homework-photo app found the hard part is not OCR but date ambiguity and task splitting — Hayk_D · 2026-07-21