MiniMind: train a 64M-param LLM from scratch in 2 hours on one RTX 3090
dr_cintas · x · 2026-11-20
MiniMind is an open-source project that lets anyone train a 64M-parameter LLM (1/2700 the size of GPT-3) from scratch in about 2 hours for roughly RMB 3 of GPU rental on a single RTX 3090.
The full pipeline covers data cleaning, pretraining, SFT, LoRA, DPO, PPO/GRPO RL, tool use, agentic RL, and distillation—all in plain PyTorch with no high-level libraries hiding the details. Extended variants include MiniMind-V (vision), MiniMind-O (multimodal omni), MiniMind-dLM (diffusion), and a linear-attention version. 85k GitHub stars; ideal for developers who've only ever called an API and want to see how an LLM is actually built.
Related event: MiniMind: Train a 64M-Parameter LLM from Scratch for ~$0.40(2 posts)→
More from Research
- Yilun Du and Dai Bo release EBM tutorial unifying diffusion, RL, and test-time reasoning — burny_tech · 2026-11-20
- 1,900 AI-for-science papers in a week: why a small test error proves almost nothing — bravo_abad · 2026-11-20
- The Future of Mech Interp: Abstracting Learned Algorithms From Circuits — burny_tech · 2026-11-20
- MBZUAI's PAN 2 World Model: 29B Params, Fully Open with Weights, Inference and Training Code — ZhitingHu · 2026-11-20
- LLMs show transferable goal encodings: a Python plan replanted into C++ gets re-implemented — paul_cal · 2026-11-20
- Solo researcher maps Qwen3.5-0.8B's 360 components as an ecosystem and repairs damaged models — suayptalha · 2026-11-20