Train your own GPT-style LLM from scratch on free Colab in about a day
Arindam_1729 · x · 2026-10-30
A GitHub repo walks you through building a transformer from scratch in pure PyTorch — no high-level model libraries — trainable on a free Google Colab in about a day.
- Architecture: attention & multi-head attention, embeddings, feed-forward layers, residual connections, layer norm
- Full pipeline: data preprocessing, tiktoken tokenization to HDF5, training with eval and LR decay, crash-safe checkpoints, text generation
- New post-training suite: Base → SFT → Reward Model → {PPO, DPO} → GRPO, all hand-written in pure PyTorch (no trl/peft/transformers), trained on Alpaca, Dolly, HH-RLHF, UltraFeedback, GSM8K, with multi-GPU DDP + bf16 support. The author has trained a 13M-parameter LLM with it.
More from coding & agent
- qwen-code v0.27.0 ships cross-platform CUA driver binaries and MCP payload filter — github-actions[bot] · 2026-10-30
- Google Knowledge Catalog goes GA with semantic model ingestion from six sources — AiJohnAllen · 2026-10-30
- Lean-checked: an AI agent improved a formalized RG bound from 10^-10^10017 to 0.00034 — _onionesque · 2026-10-30
- Codex issue reports Dot subagent losing authorizations and stalling delegated work — ingChristopherCV · 2026-10-30
- How Agno's 1,428-line Agent class stays thin: a façade pattern teardown — Mahmoud_Zalt · 2026-10-30
- Agent writes a script to remove slop comments — the script itself has 10+ lines of them — shadcn · 2026-10-30