FULL STORY

Puro-2B: Pretraining a 2B LLM on a Single RTX 5090

An arXiv paper introduced Puro-2B, a low-cost recipe for pretraining a 2B-parameter LLM on a single RTX 5090. Tsinghua's PACMAN team then open-sourced the full pipeline, drawing wide community interest.

2026-08-28 ~ 2026-09-01 · 2 episodes · 8 posts

Episode 1 · Researchers Pretrain 2B Model on a Single RTX 5090 for Under $7,000 (2026-08-28, 2 posts)

A new paper introduces Puro-2B, an open-source method that pretrains a 2B-parameter model from scratch on a single consumer RTX 5090 using FP8 precision for under $6,900, achieving performance close to Qwen2.5-1.5B.

Episode 2 · Tsinghua's Puro-2B: Open-Source Recipe Pretrains a 2B Model on a Single RTX 5090 for Under $7K (2026-08-31, 6 posts)

Tsinghua's PACMAN lab released Puro-2B, a fully open-source recipe that pretrains a 2B-parameter model from scratch on a single RTX 5090 for under $7,000, drawing wide attention as an example of algorithmic efficiency challenging brute-force compute spending.

Confirmed

  • Puro-2B is built on the Qwen2-1.5B architecture (self-styled "Poor Lab's" model), with a fully open training recipe reproducible on an RTX 5090.
  • Official cost figures: under $5,090 to match Qwen2-1.5B, or about $6,900 to approach Qwen2.5-1.5B performance.
  • The stack combines Blockwise FP8, the MuonH optimizer, curriculum model averaging, and a data recipe, training on 1.4 trillion tokens in FP8 precision.
  • Motivation: high pretraining costs make large-scale controlled experiments impractical; an open low-cost recipe enables such research.

Unconfirmed

  • Retold figures vary across reposts: @burnytech mentioned a $4,400 variant, and @kalyankpl described it as training Qwen3-1.5B on a $5,000 budget; these are second-hand retellings pending verification against the official report.

Why it matters

  • Reposters such as @burnytech read it as clever algorithms replacing millions of dollars of server cost, opening frontier pretraining to small teams and labs.
  • The fully open recipe lets the community run systematic ablations and improvements rather than merely reproducing one model.