Agents Cannot Read LinkedIn: Social Platforms Block AI Crawlers via Robots.txt
Dry_Steak30 · reddit · 2026-09-01
LinkedIn, Instagram, and TikTok block major AI crawlers (GPTBot, ClaudeBot, etc.) via robots.txt while allowing Googlebot. This prevents agents from reading user profiles directly, forcing reliance on scattered "residue" from other sites. The author highlights a challenge for person-facing agents: retrieval and browsing hit the same wall, and no standard protocol like llms.txt exists for personal data.
More from coding & agent
- LangChain adds support for MCP spec with MCPAdapter primitive — LangChain · 2026-09-01
- Claude Code supports 5 levels deep of nested subagents — airesearch12 · 2026-09-01
- Multi-Agent Data Workflow: Long-Lived Planner with Ephemeral Specialists — jeremystan · 2026-09-01
- Call for agent infra: Who will build the open source Codex-style browser? — hwchase17 · 2026-09-01
- OpenCode on pace for one billion users, open-source coding tool soars — intellectronica · 2026-09-01
- Steve Yegge's comic shows an AI agent writing its own handoff as context hits 233k — Steve_Yegge · 2026-09-01