Google publishes guide for sites to allow Google-Agent traffic, testing Web Bot Auth
OnlineInference · x · 2026-11-17
Google added an "Allow Google-Agent traffic" guide to its developer docs. Google-Agent identifies agents hosted on Google infrastructure that navigate the web and perform actions on user request.
Key points:
- Sites should explicitly allow Google-Agent in CDN/WAF/bot-detection services to avoid blocking legitimate requests
- Identification: User-Agent contains Google-Agent, and IPs come from user-triggered-agents.json
- Google is experimenting with the Web Bot Auth protocol (https://agent.bot.goog), signing requests via RFC 9421 HTTP Message Signatures; public keys are published in a .well-known directory
- An agent reaching your homepage doesn't mean it can use your site — allowing agent traffic is becoming a new technical SEO checklist item
More from coding & agent
- Mistral makes Vibe free in Europe for 3 months, claims coding edge over GLM and DeepSeek — qtnx_ · 2026-11-17
- Cheap tokens, expensive agent: Flash hits $11.97/task vs Luna's $0.05 in Databricks AI Index — krishnan · 2026-11-17
- Veteran engineer: coding AI did a tedious cmake task in 100 seconds, but only with expert guidance — AndrewOfC · 2026-11-17
- 20 Claude Code skills for falsification-gated self-improvement, grounded in FGSE paper — blaizedsouza · 2026-11-17
- Self-learning agent loop takes Honor of Kings win rate from 0% to 60% in three rounds, no retraining — kimmonismus · 2026-11-17
- User seeks a Claude Code alternative under $10/month for content and coding — Perfect_Corgi1299 · 2026-11-17