Puro-2B: Pretraining a 2B model from scratch on RTX 5090 GPUs for under $6.9K
iScienceLuvr · x · 2026-08-28
A new arXiv paper, Puro-2B, presents an open, cost-efficient pretraining recipe reproducible on consumer hardware. The authors trained from scratch on RTX 5090 GPUs with FP8 precision over 1.4T tokens — 22,514 active GPU-hours, $6.9K compute cost, 17.6 elapsed days — with the best checkpoint surpassing Qwen2-1.5B and approaching Qwen2.5-1.5B.
The paper highlights the reproduction cost barrier: training Llama-3.2-3B costs over $1.5M, and reproducing SmolLM3-3B over $700K. Efficiency comes from hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. The authors also derive a "Puro Cost Scaling Law" relating training cost to average performance, and open-source the full pipeline covering data recipe, infrastructure, software and training strategy.
Related event: Researchers Pretrain 2B Model on a Single RTX 5090 for Under $7,000(2 posts)→
More from Infra
- SK Hynix ties new US plant to looser chip equipment exports to China, sources say — pstAsiatech · 2026-10-15
- Perplexity's new embedding models: index with a 9B model, search with 0.6B — antoine_chaffin · 2026-10-15
- Unsloth ships GGUF quants for Nemotron 3.5: Bolt-74B runs in 24GB RAM — danielhanchen · 2026-10-15
- vLLM-Omni ships Day-0 serving for FLUX 3 Image Klein with BF16, FP8, NVFP4 support — vllm_project · 2026-10-15
- NVIDIA's Nemotron 3.5 Super VL lands training and inference support on Applied Compute — ypatil125 · 2026-10-15
- CoreWeave bond yield tops 16% as loans outlast customer contracts by ~2 years — tengyanAI · 2026-10-15