9.9k-star repo trains your own LLM from scratch in plain PyTorch, no trl or peft
tom_doerr · x · 2026-09-20
The GitHub repo train-llm-from-scratch (FareedKhan-dev, 9.9k stars) offers a full pipeline from raw text to an aligned, reasoning-style model.
- Hand-writes the Transformer from the Attention Is All You Need paper in plain PyTorch — no trl, peft, or transformers dependencies
- Covers everything from data downloading and pretraining to SFT/RLHF alignment; a single GPU can train million- to billion-parameter models
- Includes an sftrlhfguide.ipynb notebook and post-training docs; the core idea: turn text into numbers, predict the next token, and keep adjusting data and loss
A solid hands-on resource for developers who want to fully understand and reproduce LLM training.
More from coding & agent
- Qwen 27B one-shot prompt builds three playable Super Mario clones — EcstaticDentist · 2026-09-20
- Jev Classifies 1.6k Bookmarks in 22s, 155x Faster and 10x Cheaper Than GLM 4.7 Flash — iannuttall · 2026-09-20
- Python Is Losing Steam Because AI Doesn't Need It to Be Simple — mark_k · 2026-09-20
- Dev rips apart Musk-boosted agent's harness: 'complete garbage' Python mess — MickeySteamboat · 2026-09-20
- Databricks Co-founder: If Starting a PhD Today, I'd Work on Reward Hacking — burny_tech · 2026-09-20
- Open-source tools track what AI coding agents really spend in tokens and cost — _jaydeepkarale · 2026-09-20