Hugging Face Now Hosts Common Crawl
lhoestq · x · 2026-07-17
Shared updates note that Common Crawl, acting as the "petabytes-level shared infrastructure of the internet," is vital for open AI training.
Now that Hugging Face hosts it, users no longer need AWS credentials to access the data. This removes the need to maintain an extra secret and makes utilizing the data much more convenient.
More from Infra
- Anthropic S-1 reveals 48 cents of compute cost per dollar of Q2 revenue — nmasc_ · 2026-10-26
- Persistent AI joins UR network to route private AI inference with encrypted, IP-hidden traffic — 0xSammy · 2026-10-26
- Anthropic's disclosed compute commitments: $114.5B Google, $161.2B Broadcom top the list — BenBajarin · 2026-10-26
- SpaceX strikes $40B deal with Apollo Global to lease AI chips and data center hardware — Polymarket · 2026-10-26
- SpaceX AI revenue up 830%, Claude designs 318K antivenom proteins in 64 hours — PeterDiamandis · 2026-10-26
- Meta plans up to $95 billion stock sale to fund AI buildout — The Decoder · 2026-10-26