Train a 2B model from scratch on RTX 5090 for just $4,400
burny_tech · x · 2026-08-31
This paper presents an open-source pretraining recipe designed for cost efficiency on consumer-grade RTX 5090 GPUs. By combining hardware selection, Blockwise FP8, MuonH optimizer, and curriculum model averaging, the team trained a collection of Puro-2B models on up to 1.4 trillion tokens.
Key Findings:
- Cost Efficiency: The best model was trained for under $6,900. The derived "Puro Cost Scaling Law" suggests that only about $4,400 is needed to reach Qwen2-1.5B level performance.
- Performance: The model approaches the performance of Qwen2.5-1.5B.
- Methodology: The study demonstrates that full-stack co-design makes LLM pretraining accessible to small labs.
More from Infra
- Apple and Meta partner to run Muse Spark model on linked Mac Studios — pdamodaran · 2026-10-16
- Nvidia's developer base hits 11 million, up ~3.7x from 3 million in 2021 — Beth_Kindig · 2026-10-16
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16
- Meta launches Muse Spark Open, says it runs on two M5 Ultra Mac Studios with Apple's help — The Verge AI · 2026-10-16
- vLLM ships Day-0 LoRA RL for Muse Spark Open, lifting AIME from 0.56 to 0.85 — vllm_project · 2026-10-16