Karpathy's nanochat hits 59K stars: a full LLM training lab in one repo
techNmak · x · 2026-11-01
Andrej Karpathy's nanochat, now at 59K stars, is described as an entire LLM engineering lab in a single codebase. Unlike projects starting from pretrained weights, it ships the training pipeline itself, letting you follow the complete journey:
- Train a BPE tokenizer from raw text
- Build a Transformer in PyTorch, understanding attention, RoPE, GQA and normalization
- Pretrain on a large text dataset
- Experiment with Muon, AdamW, and mixed precision
- Fine-tune the base model for conversation
- Explore RL on specific tasks
- Evaluate and run inference with a KV cache
It also includes a single speedrun script chaining tokenizer training, pretraining, SFT, and evaluation end to end.
More from coding & agent
- VFX Supervisor Open-Sources 20 Skills for Building an Automated Motion Design Pipeline with Claude Opus 5.5 — GCWebDesigner · 2026-11-01
- Cursor launches Annotate in design mode: give agents visual prompts by editing the page — soleio · 2026-11-01
- Open-source Garmin MCP turns training data into AI-built dashboards — indaco_dev · 2026-11-01
- GPT-5.6 Sol reportedly beats Factorio, launching a rocket autonomously — Realistic-Total-3553 · 2026-11-01
- Open-source CodeRail lets teams share one Claude Code session from any device — L_i_n_k0666 · 2026-11-01
- Dev observes AI has made systems-heavy games like RTS and strategy easier to build — IanArawjo · 2026-11-01