jon_durbin's Decentralized Training Run Hits ~$11/b Tokens with Open-Sourced 1T Dataset

Developer jondurbin shared on September 11 the complete results of a decentralized model training run: roughly $11/b tokens in cost with extremely high MFU, which he called "insane" and overall "pretty legendary."

Confirmed

The training project is Pluralis-8B, trained in a decentralized fashion across 98 consumer-GPU nodes (RTX 4090/5090); its setup mirrors another project using the same infrastructure, tokenizer, and 1.3T FineWeb dataset, with Pluralis-8B serving as a comparison target.
Training cost was about $11/b tokens; jondurbin built a monitoring dashboard exposing run visibility, and MFU was excellent.
He open-sourced the pretraining dataset chutesai/mesh-1t-prod-mix: a 1-trillion-token multi-stage pretraining corpus, pretraining-only with no SFT/RL, released on Hugging Face as gated with automatic approval.
He also shared real-world data from an 8x RTX 5090 cluster doing distributed training, achieving similar $11/b tokens cost and high MFU. The choice of 5090 relates to sustained high-load scenarios, though he flagged GPU stability concerns.
Technical detail: GPU nodes simultaneously carry bandwidth load for both training and inference traffic; he reports inference performance is also strong.

Why it matters

The combination of consumer GPUs, decentralized networking, an open dataset, and public cost/MFU data offers a fully reproducible reference for low-cost open pretraining — data, monitoring methods, and hardware choices are all public.
$11/b tokens with high MFU shows the decentralized consumer-hardware route is practically viable for small-to-mid-scale pretraining, though GPU stability under sustained heavy load remains a real constraint.

2026-09-11 ~ 2026-09-11 · 5 related posts

Primary sources

Pluralis-8B decentralized training run: 98 consumer-GPU nodes at ~$11 per billion tokens — jon_durbin · 2026-09-11 01:43 (UTC+08:00)
Open 1T-token pretraining corpus released at ~$11 per billion tokens trained — jon_durbin · 2026-09-11 01:35 (UTC+08:00)
8x RTX 5090 training run hits ~$11/b tokens amid ~50% GPU failure rates — jon_durbin · 2026-09-11 01:32 (UTC+08:00)

Dev shares training dashboard: ~$11/b tokens cost with 'insane' MFU — jon_durbin · 2026-09-11
[source] 8x RTX 5090 training run hits ~$11/b tokens amid ~50% GPU failure rates — jon_durbin · 2026-09-11
[source] Open 1T-token pretraining corpus released at ~$11 per billion tokens trained — jon_durbin · 2026-09-11
[source] Pluralis-8B decentralized training run: 98 consumer-GPU nodes at ~$11 per billion tokens — jon_durbin · 2026-09-11
Training run at ~$11/b tokens with insane MFU, datasets streamed over p2p backbone — jon_durbin · 2026-09-11