Stanford CS336: A 7B model needs 14GB for weights but 84GB to train
stanfordnlp · x · 2026-10-16
Stanford NLP shared Percy Liang's CS336 Lecture 2 on PyTorch and resource accounting, with detailed notes by Chao Ma.
- Memory ledger: A 7B model needs 14GB for BF16 weights, but adding BF16 gradients and two FP32 Adam moment buffers brings the subtotal to 84GB—before activations and temp buffers. Fitting weights ≠ fitting training.
- Speed ledger: For every op ask three questions—what's stored, what arithmetic runs, what data moves. A matmul can vastly outperform elementwise ops on GPU utilization because of arithmetic intensity (FLOPs per byte moved). The companion roofline chart shows ideal BF16 GEMM on H100 SXM: with 4096×4096 weights, intensity goes from 1 at M=1 to 682.7 at M=1024, crossover at 295 FLOPs/byte, throughput bound = min(989.5 TFLOP/s, 3.35TB/s × I).
- Takeaway: Identify whether storage, arithmetic, or data movement is the binding constraint; smaller microbatches save activation memory at the cost of reuse.
More from Infra
- US data center moratoriums top 400 — an AI bull explains why they're actually good for AI — kevinsxu · 2026-10-16
- Microsoft Research unveils Magnetar, the fastest zkVM at 10-39x speedups and post-quantum security — jedisct1 · 2026-10-16
- Bonsai 2 2B hits 120 tok/s on iPhone 18 Pro, outpaces Qwen 3.5 2B at half the memory — cephaloform · 2026-10-16
- Samsung cuts Q4 phone output as Nittobo shifts E-glass to AI-chip T-glass — zephyr_z9 · 2026-10-16
- White House AI chief discusses Empire AI, New York's shared compute infrastructure — mkratsios47 · 2026-10-16
- llama.cpp merges Muse Spark Open (816B-A42B) text, vision and audio support in 4 hours — jacek2023 · 2026-10-16