Stanford CS336: Building Transformer From Scratch
stanfordnlp · x · 2026-07-20
This image showcases Assignment 1 of Stanford CS336, themed "Building a Transformer Language Model from Scratch." The assignment requires students to implement a BPE tokenizer, Transformer LM, cross-entropy loss, and AdamW optimizer, along with training loops and saving/loading of model and optimizer states. It also requires training the tokenizer and model on TinyStories, performing sample generation and perplexity evaluation, and further training on OpenWebText to submit perplexity scores to a leaderboard. This indicates the course emphasizes a complete model training pipeline rather than just calling existing libraries.
More from Research
- AI tools are changing counterexample hunting in formalized mathematics — burny_tech · 2026-07-21
- The Empirical Flaws in Mechanistic Interpretability Research — burny_tech · 2026-07-21
- Generative Bionics turns its Gene.01 humanoid from CES concept into a walking platform — CyberRobooo · 2026-07-21
- Immune foundation model MAESTRO aims to predict future immune trajectories — arjunrajlab · 2026-07-21
- Safe RL paper on directional constraints accepted to IROS 2026 — Jan_R_Peters · 2026-07-21
- Graph engineering is emerging as the next step after loop-based AI workflows — TheTuringPost · 2026-07-21