Dev Open-Sources AEO Audit Engine to Detect AI Crawler Blockages
goldlionroar · reddit · 2026-08-05
An indie developer open-sourced their AEO (AI Engine Optimization) audit engine to help websites troubleshoot crawling issues with LLM bots.
Core Pain Points & Features:
- The developer noticed that while a site's robots.txt might look perfectly clean, CDNs often block bots like GPTBot, returning 403 errors.
- This open-source tool (MIT license) fetches pages simulating 9 major AI bots (like GPTBot, ClaudeBot), scores 5 retrieval gates and 8 headline checks, and generates fix files.
- It can be integrated as an MCP server into tools like Claude Code or Cursor, allowing agents to automatically write fixes into the repo.
Business Insight:
For indie devs looking to land clients, running this audit on a prospect before sending a cold email serves as a highly effective icebreaker compared to sending an empty pitch.
More from coding & agent
- Developer Waited 6 Months for Claude, Built Entire App in One Day — airkatakana · 2026-08-05
- Proposed cmux Multi-Column Sidebar: Machines, Workspaces, and Agents — philipvollet · 2026-08-05
- Chinese Models Dominate Forecasting Leaderboard with Advanced AI Agents — teortaxesTex · 2026-08-05
- eidoverse-worlds: Open-Source Persistent 3D World Where Humans and AIs Co-Exist — repligate · 2026-08-05
- AARM Spec for AI Agent Audit Trails Released: 9 Properties to Fight Memory Poisoning — Funky_Chicken_22 · 2026-08-05
- What is a 'Software Factory'? AI Agents Reshape the SDLC — Pavan_Belagatti · 2026-08-05