Developer on learning LLMs by building from scratch with Raschka's book
rasbt · x · 2026-11-12
Developer brianreyx says building things from scratch taught him the most about how LLMs work, and he's currently working through Sebastian Raschka's Build a Large Language Model (From Scratch).
His study notes show the book's explanation that self-attention has no notion of token position and the embedding layer maps the same token ID to the same vector regardless of location, so positional information must be injected separately — alongside a Jupyter notebook combining token and position embeddings.
More from Companies & People
- Anthropic Gives CNCF Maintainers 3,000 Free Claude Ultra Codes — HowDevelop · 2026-11-12
- Georgia Tech's Molei Tao Recruiting PhD Students and Postdocs in Diffusion Language Models — gautamcgoel · 2026-11-12
- YC's Jessica Livingston revisits Theranos: the original whistleblower Tyler Shultz — ycombinator · 2026-11-12
- Meta open-sources Astryx: 8-year design system with 150+ React components and an agent CLI — techNmak · 2026-11-12
- Google engineer confirms internal use of Gemini for coding: 'Googlers are good judges' — zacharynado · 2026-11-12
- Vercel CEO Rauch: great products still demand extraordinary focus and love — santiviquez · 2026-11-12