rasbt's Reasoning From Scratch Finale: Distilling Reasoning Models Hands-On
rasbt · x · 2026-10-17
Sebastian Raschka's "Reasoning From Scratch" series concludes with lecture 8, focused on distillation as an efficient alternative to the prior RL training runs.
The lecture implements the full pipeline from scratch: soft vs. hard distillation, on-policy vs. off-policy tradeoffs, preparing data from the DeepSeek R1 MATH dataset, inspecting reasoning traces, tokenization and chat templates, handling the 2,048-token sequence limit, computing answer-token log probabilities and cross-entropy loss with a Qwen3-0.6B base model, running the training loop, and finally evaluating checkpoints on MATH-500 and comparing teachers. Future directions and exercises close out the series.
Related event: Raschka Concludes 'Reasoning From Scratch' Series with Distillation Finale(2 posts)→
More from Research
- Natural stimuli make V1 responses dirty via population effects — HistedLab · 2026-10-17
- Neuroscientist: strong polysemanticity found in nearly every brain area — HistedLab · 2026-10-17
- Google.org picks 17 organizations for $36M AI for Science Impact Challenge — ymatias · 2026-10-17
- Romik's ambidextrous sofa optimality formalized and proven in Lean 4 — IgorCarron · 2026-10-17
- AstraZeneca strikes $40 million AI drug discovery deal with insitro — HealthcareLdr · 2026-10-17
- YC Paper Club: Stanford AI Safety Directors on OpenAI-Hugging Face Incident and an NTSB for AI — Y Combinator · 2026-10-17