Google Explains robots.txt Rules for AI Crawlers, Emojis, International URLs
OnlineInference · x · 2026-10-22
Google's Gary Illyes and Chery Prommawin released the second video in their Non-Canonical Cabinet series on what robots.txt actually supports. Key takeaways: robots.txt governs Googlebot crawling and should follow RFC 9309; non-English characters in rule paths need percent-encoding; don't assume a human-readable rule works — validate syntax and test URL patterns; be careful with inconsistent encoding of international characters and emojis. Google-supported directives may not be honored by other search engines or AI crawlers.
More from Safety
- 136-Country, 118M-Article Dataset Audits State Media Share in LLM Training Corpora — deliprao · 2026-10-22
- PHP 8.5.12 may still be vulnerable to php2shell, researchers report — basedjensen · 2026-10-22
- Cloudflare's security-audit-skill hits 35.7k stars, turns coding agents into auditors — Pytlicek · 2026-10-22
- Wiz: stolen credentials from LiteLLM supply chain incident already abused for LLMJacking from two tracked IPs — wunderwuzzi23 · 2026-10-22
- The Supply Chain Dilemma: Patch Fast for Vulnerabilities, Slow for Backdoors — jd_pressman · 2026-10-22
- Security apocalypse is ultimately a test of whether people are morally good — moultano · 2026-10-22