Open-licensed training data keeps growing: the 8TB Common Pile v0.1
BlancheMinerva · x · 2026-10-17
EleutherAI's Stella Biderman pushes back on data-exhaustion fears: even as sites restrict crawling, openly licensed and public domain data keeps growing because more data is created each year. She links the Common Pile v0.1 paper — an 8TB openly licensed pretraining corpus spanning 30 sources (papers, code, books, encyclopedias, educational material). The team validated it by training two 7B models (Comma v0.1-1T/2T) that match models trained on unlicensed text at similar compute. Roughly half the corpus (3.8TB) was created after 2020.
More from Safety
- Perplexity CEO slams X's 'aesthetic standard' for ads while AI hate images of Indians persist — rbhar90 · 2026-10-17
- P-256's Birth Moment Reconstructed: Friday Aug 15, 1997, 9:52PM at Fort Meade — matthew_d_green · 2026-10-17
- Claude Cracks Decades-Old Crypto Mystery: Origins of NIST P-256 Curve Seeds Revealed — matthew_d_green · 2026-10-17
- Open-weight cyber capabilities just 6–7 months behind frontier, GLM-5.3 beats GPT-5.4 — Jsevillamol · 2026-10-17
- Anthropic hires Trump's former Deputy DNI as registered AI lobbyist, filing shows — ShakeelHashim · 2026-10-17
- Most database MCP servers share one connection string — an engineer adds per-query authorization — Then_Respect_1964 · 2026-10-17