Stanford CS336 Student Implements Tokenizer From Scratch, Shares Takeaways
stanfordnlp · x · 2026-10-02
A student following Stanford NLP's CS336 class — which teaches language modeling from scratch — has implemented the pre-tokenizer and tokenizer themselves and shared their takeaways in a thread. CS336 is Stanford's open course for building LMs from the ground up.
More from Models
- Why ChatGPT keeps answering 47 or 73 when asked for a number — Princevora03 · 2026-10-02
- Daily brief: Grok 4.7 rolls out as base model everywhere, Claude Code ships Mods — testingcatalog · 2026-10-02
- Screen Studio trained on 5,000 fake macOS apps to build a UI-specific upscaler — CurieuxExplorer · 2026-10-02
- Intelligence is jagged: use the cheapest, lowest-latency model that's good enough — sachdh · 2026-10-02
- Gemini 4 may roll out to Ultra users next week as Google's TPU capacity reportedly runs tight — haider1 · 2026-10-02
- Barred from Qwen/DeepSeek at Work, Redditor Seeks Sub-40B Local Models for Coding and Long-Doc QA — AdRepulsive7837 · 2026-10-02