Open-source repo trains a Transformer LLM from scratch — 13M params on a single Colab T4
Roger_M_Taylor · x · 2026-09-19
A step-by-step open-source repo walks you through building and training a Transformer from scratch in PyTorch, following the original "Attention is All You Need" paper: MLP, single-head and multi-head attention, transformer blocks, and the full model, each coded and explained with detailed diagrams.
Training data comes from The Pile, an 825GB open-source dataset spanning books, articles, code and websites, with scripts for downloading, preprocessing and tokenizing via tiktoken, storing in HDF5, and feeding training batches. The author demonstrates training a 13M-parameter model on a single Colab T4 GPU, at which point the model starts producing proper grammar and coherent short sentences.
More from coding & agent
- Readback: free MIT VS Code extension reads Claude Code replies aloud via Speechify — shauntrennery · 2026-09-19
- GitHub Next open-sources LocalJev, a local Jev-compatible API built on oMLX and DiffusionGemma — gaganghotra_ · 2026-09-19
- WebMCP benchmark: Jev + Mercury 2.5 solves 100% of tasks at 112x lower cost than GPT-6 Astra — hardimanjames · 2026-09-19
- Musecases Launches: A Community-Voted Prompt Library for AI Agents — ChrisUniverse · 2026-09-19
- ~40,000 passing tests: dev explains why he doesn't review every line of AI code — doodlestein · 2026-09-19
- NVIDIA's SoL-Pi GitHub repo: auto-research loops for efficient agent harnesses — aigclink · 2026-09-19