LLMs-from-scratch Repository Surpasses 100k GitHub Stars
rasbt · x · 2026-08-07
Sebastian Raschka announced that his open-source tutorial project LLMs-from-scratch has surpassed 100,000 stars on GitHub. The project is designed to help developers understand and build large language models from the ground up.
Key highlights of the repository include:
- Complete From-Scratch Code Path: Covers tokenization, attention mechanisms, pretraining, classification, and instruction fine-tuning.
- Mainstream Architecture Implementations: From-scratch implementations of smaller variants of Llama, Qwen, Gemma, and Olmo that run locally.
- Cutting-Edge Techniques: Includes implementations of GQA, MLA, sliding-window attention, DeepSeek Sparse Attention, cross-layer KV sharing, and MoE.
- Engineering Practices: Covers KV caching, training performance, memory-efficient weight loading, DPO, evaluation, and LoRA.
Raschka also teased a larger applied custom “small” LLM project he is currently working on, promising to share more soon.
Related event: "LLMs-from-scratch" Open-Source Tutorial Surpasses 100k GitHub Stars(2 posts)→
More from Research
- AI will be superhuman at science regardless, and Europe can't opt out — teortaxesTex · 2026-10-16
- SERO augments long-horizon agentic RL with executable oracle continuations — Shanghai-AI-Laboratory · 2026-10-16
- Tsinghua and Amap propose OmniVerifier-C1, a graded verifier beating binary judges — Bowei Liu · 2026-10-16
- Alibaba open-sources MAI-Gym, an all-in-one GUI agent eval and RL training platform — Tongyi-MAI · 2026-10-16
- Peking University's SimForcing distills simulation motion priors into robot world models — PekingUniversity · 2026-10-16
- CoWorld Reasoner uses an evolving video world model as a visual reasoning engine — ModalityDance · 2026-10-16