From-scratch 8M-parameter LLM: dev blogs the whole training journey
the-random-guy-2002 · reddit · 2026-11-01
A developer built and trained a tiny 8M-parameter LLM from scratch in PyTorch as a lab for implementing ideas from research papers, documenting everything in candid blog posts.
So far: one post on building a Transformer without tutorials, another on 60,000 training steps, hitting a loss plateau, and the simple fix that resolved it. Next up: KV caching, PagedAttention, and Orca. The post (punctuated with an "I am ready to get hurt again" meme) asks the community whether such experiment writeups are welcome — purely for learning and feedback, not promotion.
More from Research
- New talk: How to retarget human behavior to humanoid robots — notmahi · 2026-11-01
- CrowdGPT v2.6: crowdsourced LLM training adds 4.5B tokens in 6 days — Vxtzq1 · 2026-11-01
- AI agent performs first end-to-end epilepsy treatment in a mouse, from gene therapy design to robotic brain surgery — Distinct-Question-16 · 2026-11-01
- Interactive Explainer Demystifies Gated DeltaNet, the Fixed-State Linear Attention Behind Recent Models — CatAstro_Piyush · 2026-11-01
- BooBench: A Halloween Benchmark That Scares LLMs and OCR Tools Into Character-Repeating Meltdowns — xennygrimmato_ · 2026-11-01
- Interactive visualizer lets you zoom into Esperanto's ET-SoC-1 AI chip, down to FMA instructions — yaroslavvb · 2026-11-01