Sentry founder: my well-behaved AI crawler gets blanket-blocked by default
Joseph_myke · x · 2026-09-08
Sentry founder zeeg complains that his throttled, well-behaved web crawler is banned by default on many sites, while repliers back the "responsible citizen" approach and criticize sites that blanket-block bots on fully public pages — a snapshot of the AI-era crawler standoff.
Related event: Sentry CEO Complains Well-Behaved AI Crawler Still Gets Blocked(3 posts)→
More from Infra
- Both Nvidia and Google run ARC synthetic data generators — GregKamradt · 2026-09-08
- Memory capex to hit $97B in 2026, $146B in 2027 — 56% of next year's semiconductor spend — Beth_Kindig · 2026-09-08
- TokenSpeed stabilizes Kimi K3 on ~1,000 B300 GPUs for 3+ weeks: GB300 TP8 optimization details — zhyncs42 · 2026-09-08
- KV Cache Math From Scratch: A Deep Workshop on LLM Inference at Scale — AI Engineer · 2026-09-08
- Intel extends High NA EUV lead as TSMC and Samsung confirm adoption by end of decade — BenBajarin · 2026-09-08
- User seeks a custom GGUF quant to fit GLM on a 192 GB RAM Mac between Q2 and Q4 — CentrifugalMalaise · 2026-09-08