ML researcher slams Bright Data: rejects robots.txt pages for most, but lifts blocks for paying clients
burkov · x · 2026-09-30
ML researcher Andriy Burkov called out Bright Data's proxy service: its pricey Browser API is detectable most of the time, and it rejects requests to pages disallowed by robots.txt — yet grants exceptions to select clients, who are invited to "contact your sales rep" to lift the block. He asks the community for honest proxy services that actually work against rate limits and IP blocks.
More from Companies & People
- AI launches feel like obvious follow-ons: users ignored while real usage stays abysmal — vaibhavbetter · 2026-09-30
- Full OpenAI DevDay 2025 Video With Chinese-English Subtitles Now Available — xiaohu · 2026-09-30
- repligate: Lighthaven Events Never Felt Uncomfortable, Sexual or Sexist to Me — repligate · 2026-09-30
- Founder Akshaya Dinesh joins xAI's product team after building with Cursor and Grok — katieruthmishra · 2026-09-30
- AT&T CEO says SpaceX's phone strategy is not viable: "Satellite won't beat fiber" — RachelVT42 · 2026-09-30
- Wayve's supervised robotaxi ride in London via Uber impresses, closing in on Waymo — Sethwinterroth · 2026-09-30