LLMs-from-scratch Repository Surpasses 100k GitHub Stars
rasbt · x · 2026-08-07
Sebastian Raschka announced that his open-source tutorial project LLMs-from-scratch has surpassed 100,000 stars on GitHub. The project is designed to help developers understand and build large language models from the ground up.
Key highlights of the repository include:
- Complete From-Scratch Code Path: Covers tokenization, attention mechanisms, pretraining, classification, and instruction fine-tuning.
- Mainstream Architecture Implementations: From-scratch implementations of smaller variants of Llama, Qwen, Gemma, and Olmo that run locally.
- Cutting-Edge Techniques: Includes implementations of GQA, MLA, sliding-window attention, DeepSeek Sparse Attention, cross-layer KV sharing, and MoE.
- Engineering Practices: Covers KV caching, training performance, memory-efficient weight loading, DPO, evaluation, and LoRA.
Raschka also teased a larger applied custom “small” LLM project he is currently working on, promising to share more soon.
More from Research
- Pretraining Potential: Minimal SFT on Reasoning Traces Significantly Boosts LLM Thinking — antirez · 2026-08-07
- Free Open-Source Visual Masterclass: 12 Chapters on LLMs — mdancho84 · 2026-08-07
- Next-Gen Science Labs Built on World Models and Agent Layer — MengdiWang10 · 2026-08-07
- Stanford's Evo 2 AI Tool Designs Novel Bacteriophage to Kill E. coli — HumbleRestaurant790 · 2026-08-07
- Quanta Deep Dive: Why AI is Cracking the Legendary Erdős Math Problems — theomitsa · 2026-08-07
- Scale AI Paper Proposes Interaction-Centric Taxonomy to Localize Agent Failures — theomitsa · 2026-08-07