Google's Anti-Scraping /goto Links Backfire: Getting Indexed Instead
lilyraynyc · x · 2026-08-28
Google's anti-scraping measure of using encrypted google.com/goto redirect links has backfired: these URLs are now being indexed by Google itself. Since robots.txt disallows crawling /goto, Googlebot cannot see the redirect, potentially causing the original web pages to be cannibalized by these Google-domain links in search results.
More from Infra
- ASML mirror supply becomes the new AI compute bottleneck — TheZvi · 2026-08-28
- NVIDIA Details QAD Pipeline for Optimizing Nemotron Model — PyTorch · 2026-08-28
- Developer Runs 291B Model Locally on Four Mac Studios — eptwts · 2026-08-28
- Harness launches AI code repository designed for high-volume Agent commits — rseroter · 2026-08-28
- Cloudflare opens monetization gateway for APIs and MCP tools — kleffew94 · 2026-08-28
- Survey: 72% of Devs Prefer 5x Speed Over 20% Smarter Models — sonic_op · 2026-08-28