Indie pretraining run hits 10T tokens of data, locks in Apple M5 and DGX Spark support
jon_durbin · x · 2026-11-05
Developer jondurbin shared progress on his independent pretraining effort, arguing that post-midterm uncertainty over US datacenter capacity makes compute diversification non-optional.
- Dataset prep nearly done: 10T tokens of high-quality data tokenized and upsampled
- Apple M5 and NVIDIA DGX Spark support locked in; AMD Ryzen AI MAX+ PRO 495 support pending hardware arrival
- Settled on the openai o200k tokenizer, modified to always use single tokens for digits to improve math capability
- The code/agentic pipeline is in final touches but non-blocking for pretraining
He cites Buckminster Fuller: you change things by making them obsolete through superior technology.
More from Infra
- Beff backs space data centers: policy will only accelerate the Dyson swarm — beffjezos · 2026-11-05
- Texas datacenter moratorium sparks backlash over trillions in lost investment — beffjezos · 2026-11-05
- Researchers praise Together Compiler for open discussion of self-improving infra tradeoffs — simran_s_arora · 2026-11-05
- The compute arbitrage math: every $1 in data centers returns $4 — TinfoilTricorn · 2026-11-05
- Unsloth ships Qwen3.8-Flash-Next local guide: 125B MoE runs on 75GB RAM, 1-bit is 79% smaller — gnukeith · 2026-11-05
- Ripple ladder shows data beats architecture: 4x gains from new dataset vs 3x from new arch — finbarrtimbers · 2026-11-05