News site logs show AI bots hitting it hundreds of thousands of times a day
PJZNY · reddit · 2026-09-26
A mid-level news site shared its daily logs: AI bots scraped the site hundreds of thousands of times in a single day to pull information for their answers. The poster frames this as a concrete picture of why AI needs massive crawling infrastructure — and points toward a future where AI systems may have to pay creators for all this content usage.
More from AGI Musings
- Drug scientist: AI speeds up medicine, but the benefits stop at those who can pay — realmeetjames · 2026-09-27
- Mars Society guest essay: how AI could help humanity explore and settle space — EdwardHeisler · 2026-09-27
- Give AI tools a real shot and you'll find your 'unique skills' are now solved — dreamwieber · 2026-09-27
- Alignment researcher: novel pretraining + strong agentic RL is where >90% of AI risk concentrates — menhguin · 2026-09-27
- Alex Zhang on LM 'shape': redesigning models around harnesses, not the other way — a1zhang · 2026-09-26
- AI needs more compute rails: why blocking data centers in poor regions backfires — xiaosun86 · 2026-09-26