Open 1T-token pretraining corpus released at ~$11 per billion tokens trained
jon_durbin · x · 2026-09-11
Developer jondurbin published the full dataset behind his pretraining run, chutesai/mesh-1t-prod-mix: a trillion-ish-token multi-phase pretraining corpus (no SFT/RL, pretraining only), gated with auto-approval on Hugging Face.
- Stored as flat llama3 uint32 token bins (EOS 128001 separators) with val bins, per-phase mixmanifest. / poolconfig. provenance, and SHA256SUMS
- Phase budgets include 600B tokens (ClimbMix+FinePDFs-Edu, UFW, Stack-code, CC-Math, etc.) and a 285B second phase
- Training cost ran $11 per billion tokens, which he calls insane; MFU also impressively high
- Uses a few nodes from lium.io; a run dashboard is available
More from Infra
- DeepSpeed shows AutoEP/AutoTP/AutoSP parallelism gains, hosts office hours — PyTorch · 2026-10-27
- Running a local 9B as a coding agent: brief the big model to plan, let the small model execute — Electrical_Ninja3805 · 2026-10-27
- Cloudflare Durable Objects validated as top architecture for AI agent WebSocket servers — tristanbob · 2026-10-27
- Gary Marcus: With Q3 data added, Anthropic's compute-efficiency growth shows deceleration — GaryMarcus · 2026-10-27
- SpaceX locks in $40B GPU financing deal with Apollo, the largest ever committed — TechCrunch AI · 2026-10-27
- Inference is just Kubernetes with a multi-trillion-dollar market, and the communities overlap — matt_slotnick · 2026-10-27