Tsinghua's PuRo-2B trains from scratch on RTX 5090s for $6.9K and beats Qwen2-1.5B

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

Kairong Luo, Jiarui Cui, Yaorui Yin, Shengqi Chen, Yiming Yang, Linxiang Gao, Yanmohan Wang, Mingzhe Zhang, Kaiyue Wen, Kaifeng Lyu, Wenguang Chen

cs.CL, cs.LG

2026-08-28

From-scratch ~2B on 1.4T tokens via RTX 5090s, FP8 and MuonH costs $6,891 GPU-side and scores 57.81 on 15 tasks, above Qwen2-1.5B (55.14), below Qwen2.5-1.5B (60.73).

What problem this solves

Open weights are common. Fully reconstructible pretraining still tends to be expensive. Under this paper's rental-equivalent ledger, Llama-3.2-3B sits above $1.5M, SmolLM3-3B around $719K, and OLMoE-1B-7B around $200K. A public recipe can still be out of reach for a small lab.

Tsinghua PACMAN and Pengcheng Laboratory publish a from-scratch recipe aimed at that gap and train a family of 2B PuRo-2B (普罗-2B) checkpoints on consumer RTX 5090 GPUs. The comparison anchors are Qwen2-1.5B and Qwen2.5-1.5B. Weights, data manifests, Megatron training code, and Kaiyuan-Spark processing are released under Apache 2.0; upstream data licenses stay component-specific.

Method

The backbone follows the Qwen3-1.7B layout with untied embeddings and LM head, which lands near 2B parameters. Sequence length is 4,096 and the global batch is 1,536 sequences. Phase 1 runs 438.8B tokens on 24 GPUs. Phase 2 continues for 960B tokens on 96 GPUs. The scheduled total is about 1.4T tokens.

Five pieces share the savings, and they were co-designed rather than stacked blindly.

• Hardware is the RTX 5090. On their peak-compute-per-dollar proxy, BF16 and FP8 are about 2.7× an H200, at an effective $0.31 per GPU-hour. Memory is 32 GB and there is no NVLink. NVIDIA disables PCIe P2P and GPUDirect RDMA on consumer cards; a patched open driver raised one-way P2P from 31.5 to 56 GB/s and cut latency from 14.3 μs to 0.4 μs, while 24-GPU AllReduce busbw moved from about 8.87 to 19.93 GB/s. That change is unsupported and may conflict with the EULA. The stock driver still trains, only slower.
• Transformer linear GEMMs use blockwise FP8 from random init (128-wide activation groups, 128×128 weight blocks, MXFP8 on Blackwell). Master weights and optimizer state stay BF16/FP32. Versus BF16, validation loss rises 0.0031 to 0.0039, about 98% compute retention. At 1.7B, throughput is 1.36× and the quality-adjusted net is about 1.34×.
• Approximately scale-invariant attention and MLP matrices use MuonH (Muon plus Hyperball: normalize the update, then project back to the initial Frobenius radius). Everything else uses AdamW, with a 10× learning-rate multiplier on the Hyperball group. Versus plain Muon, matched validation loss needs about 16% less theoretical compute.
• Phase 2 Curriculum Model Averaging sorts each source by its own quality scores, low to high, into 376 mixture-preserving buckets. Late training resumes from step 218,000 at a frozen learning rate and averages six checkpoints.
• Data sources and mixture weights come from small proxy runs, not a single cross-domain quality score. Phase 2 drops English to 59.4%, raises math to 18.3% and code to 11.5%, and adds 1.3% instruction-style text.

There is no tensor parallelism. Phase 1 uses PP=2 and DP=12 and reports about 73% mixed-precision MFU.

Results

The dollar figure counts only active GPU-hours of the final two-phase run. It excludes data prep, proxy sweeps, failed jobs, post-training, and labor. The canonical run is 22,514 GPU-hours and $6,891. A uniform Phase 2 variant that already clears Qwen2 uses 14,262 GPU-hours and $4,370.

On the unweighted 15-task mean, PuRo-2B scores 57.81, Qwen2-1.5B 55.14, Qwen2.5-1.5B 60.73. The Qwen dollar values on the same plot are 6ND plus H100-equivalent estimates (about $84K and $217K), so they are not the same kind of invoice as the measured 5090 hours.

The four-task math and code average is 43.50, 3.21 above Qwen2 and 4.02 below Qwen2.5. The eleven-task reasoning set is 63.02, 2.48 above Qwen2 and 2.51 below Qwen2.5. Under the same protocol the model still trails SmolLM3-3B (65.85) and Qwen3-1.7B-Base (65.27); those estimated reproduction costs are one to two orders of magnitude higher.

Curriculum ordering beats a uniform shuffle by about 1.2 to 1.6 points on the 15-task mean. The gap survives SFT: focused-math GSM8K 68.66 vs 66.89, scaled-math 76.12 vs 74.10, broad instruction 56.58 vs 54.99.

The fitted Puro Cost Scaling Law is a 2B, recipe-specific scale-down curve. Inverting it puts the Qwen2 aggregate near $4.4K.

Why it matters

For a group that actually wants to rerun pretraining, this is a rare package of consumer GPUs, trillion-token scale, an inspectable recipe, and an explicit dollar boundary. The 5090 compute-per-dollar edge is real. Hitting the reported throughput still needs driver patches, pipeline layout work, and a willingness to live with 32 GB.

"Approaches Qwen2.5" means a roughly 3-point gap on this 15-task protocol, not a tie. The $5,090 in the title is the claim that matching Qwen2-1.5B can fit under that number on their scale-down fit.

Limitations

The cost is marginal accelerator spend, not the bill for standing up the lab. Many comparator dollars are 6ND estimates, so the x-axis is not a price list. Most of the mixture is already-filtered public data, and there is no corpus-wide decontamination audit against the evals; some of the curriculum gain could be leakage. The authors point to the post-training gap as evidence the ordering still helps, but the audit is missing. Chinese tokens are in the mix and then left out of the headline tables. The run is an overtrained 700 tokens-per-parameter dense model, not compute-optimal, and the cost law does not extend to larger models. 32 GB and no NVLink will bite first if the architecture grows.

Terms

• Reproduction-cost boundary: GPU-hours of the final Phase 1+2 run times a rental rate; data engineering and exploratory runs are out of scope.
• MuonH: Muon with a Hyperball projection that pins each matrix to its initial Frobenius radius so the effective learning rate is the scheduled one.
• Curriculum Model Averaging: within-source low-to-high ordering, then a constant-LR tail and an average of late checkpoints.
• blockwise FP8: per-block scales, so a few outliers do not stretch the quantization grid of a whole tensor.
• MFU: achieved model FLOP/s over peak, here weighted by the FP8/BF16 mix of the work.

Source

• arXiv 2608.27370v1
• PDF
• Hugging Face

What people are saying

• Puro-2B: Pretraining a 2B model from scratch on RTX 5090 GPUs for under $6.9K iScienceLuvr

Related papers

• Apple Agent Seer: MCP-spec eval harnesses score 0.911, argument values still fail
• Agent environment learning follows a log-sigmoid (R²=0.998); speed doubles every 3 months
• CritICL Turns Small-Model Failures into One-Pass ICL That Matches Consistency@5
• GSQ Closes Most of the 2-bit Gap to QTIP While Staying on Scalar Kernels
• Asymmetric TTPO Lifts Qwen3-1.7B from 38.0% to 45.2% With No Labels
• CaRGo-T turns humor into a causal graph, with up to ~20% understanding gains

All paper explainers