Tsinghua's open-source Puro-2B beats Qwen2-1.5B with just $4,400 of pretraining on RTX 5090s
新智元 · wechat · 2026-10-15
Tsinghua's Puro-2B is a fully open-source "budget" pretraining recipe — data, framework, configs, and weights — trained from scratch on consumer RTX 5090 clusters.
- 2B params, up to 1.4T tokens: $4,400 compute beats Qwen2-1.5B; a $6,900 run approaches Qwen2.5-1.5B
- Five-part co-design: RTX 5090s deliver 2.77x BF16 compute per dollar vs H200, with PCIe P2P and GPUDirect RDMA tuning pushing MFU above 70%
- FP8 training (1.34x net efficiency) and the MuonH optimizer with explicit effective-LR control (1.19x)
- Data curriculum + Curriculum Model Averaging (CMA) yields up to 2.4x cost efficiency; PuroCostScalingLaw estimates performance before training
- Fine-tuning experiments confirm CMA's pretraining gains carry through post-training (GSM8K +1.77-2.02 pts)
More than a leaderboard score, it's an affordable, reproducible pipeline for pretraining research.
More from Infra
- IFM open-sources RL360: 8,192 concurrent sandboxes and ~1.3M rollouts for agentic RL — rupspace · 2026-10-23
- SpaceX seeks access to 80,000 US cell tower sites; Polymarket puts carrier odds at 29% — Polymarket · 2026-10-23
- Modular open-sources MAX framework to free AI compute from single-vendor lock-in — clattner_llvm · 2026-10-23
- Claude Models on Google Cloud (Anthropic managed) is now generally available — claudeai · 2026-10-23
- Scaling GLM-5.3-Flash training to 1M token context, 64x over Megatron baseline — AndrewLBeam · 2026-10-23
- Perplexity's Portable Computer lands on AMD Ryzen AI Max, running local 27B model — AravSrinivas · 2026-10-23