Screaming Frog tip: your AI assistant can fully configure your SEO crawler
VeryWellVersed · x · 2026-11-10
- Screaming Frog's official tip: an AI assistant can fully adjust your crawler config via the SEO Spider MCP — from basics like enabling JavaScript rendering or switching user-agent, to advanced setups like custom extraction, complex segments, and custom JS
- The screenshot shows a real workflow: asked to extract author names and publish dates from blog posts and crawl a site, Claude set up two XPath extractors, chose meta tags over visible bylines (single clean values, ISO-sortable dates, blank on non-posts), flagged a fallback XPath for co-authored posts, and handled three housekeeping steps
More from coding & agent
- turbopuffer ships CJK tokenizers; dev shares hard-won Chinese/Japanese/Korean search tuning tips — bo_wangbo · 2026-11-11
- Adobe Firefly Builder (beta): describe a website, get a live editable React app — rufusd · 2026-11-11
- Zero-dependency MCP hub fronts nine research repos as 14 tools without merging dependency trees — Emergency-Tutor-7 · 2026-11-11
- Video Decompiler: Open-Source Skill Has Fable 5.2 Code Every Frame of a Video — brandon_galang · 2026-11-11
- Gemini 4 Carbon tested: 100+ unattended passes rebuild a 3D fort in a raw engine — Prompt Engineering · 2026-11-11
- 53% Less Code: Ponytail Forces Agents to Ask 'Does This Need to Exist?' Before Writing — nevrekaraishwa2 · 2026-11-11