Confirmed: AI assistant Instinct fetches web pages via Exa, with signed Web Bot Auth requests
gaganghotra_ · x · 2026-10-17
Independent testing confirmed that the AI assistant Instinct uses Exa for web search and page fetching. The tester planted URLs for nonexistent pages on a site they administer; since requests targeted fake pages and timing matched their conversation with Instinct, the source was unambiguous — about 12 requests over 90 minutes.
Key findings from Cloudflare analytics and raw server logs:
- Every direct fetch arrived with the user agent Mozilla/5.0 (compatible; ExaUser/1.0), each time from a different IP
- Exa's ExaUser fetcher signs every request with Web Bot Auth (RFC 9421), automatically verifiable at Cloudflare, Akamai, and others
- Per Exa's docs, ExaUser only fetches pages on demand (not automatic crawling, robots.txt may not apply), accesses public content only, and does not bypass logins, paywalls, or CAPTCHAs
More from coding & agent
- StackOne ships Tool Policies: fine-grained guardrails for MCP tool calls — afp-media · 2026-10-17
- Huntley: developers displaced by AI 'failed a curiosity test' — rseroter · 2026-10-17
- Teaching may be the best prerequisite for prompting agents, argues Geoffrey Litt — sh_reya · 2026-10-17
- RAX open-sources everything: zero demos, all experiment sessions included — du_yilun · 2026-10-17
- RAX separates the coding agent workspace from the robot stack for multi-embodiment — du_yilun · 2026-10-17
- shadcn reflects: it's only been a year since Claude Skills launched — shadcn · 2026-10-17