From-scratch GPT-2 124M pretrained on 2.48B tokens with 4x A40 — full writeup
upulbandara · reddit · 2026-10-20
To understand LLM pre-training hands-on, the author built and trained GPT-2 (124M params) from scratch: 2.48B tokens (20 tokens per parameter) on 4 NVIDIA A40 GPUs, producing reasonably coherent English output. Inspired by Karpathy's nanoGPT work, the code and training details are open-sourced on GitHub (upul/Build-GPT-2).
The project covers the key pre-training pieces:
- Building a GPT-style Transformer architecture
- Preparing and tokenizing large-scale datasets
- Distributed training with PyTorch DDP
- Mixed-precision training and gradient accumulation
- Learning-rate scheduling and optimization
- Evaluation, checkpointing, and experiment tracking
Next step: a 1B model with RoPE, RMSNorm, SwiGLU, and Grouped-Query Attention.
More from Infra
- Chrome 155 ships JPEG XL: 30-50% smaller than JPEG with lossless JPEG transcode — banteg · 2026-10-20
- Polymarket: DeepSeek releases open-source tool to sharply cut AI inference compute costs — Polymarket · 2026-10-20
- Together, Fireworks, Prime Intellect and Modal are all converging on inference + RL training — NielsRogge · 2026-10-20
- NVIDIA GTC kicks off in Berlin with Jensen Huang keynote set for tomorrow — NVIDIAAI · 2026-10-20
- Hyperdrives CEO interview: cheaper power-dense electric motors via cooling breakthrough — MarvinTBaumann · 2026-10-20
- MiniMax H3 video generation runs on a single RTX 3060 12GB via GGUF quantization — Ok-Inevitable8391 · 2026-10-20