New video explains how to train models too big for one GPU: ZeRO and FSDP demystified
jbhuang0604 · x · 2026-10-19
Prof. Jun Huang (CUHK) released a new tutorial video, 'How to Train Models Too Big for One GPU,' covering:
- How real numbers are represented and stored in memory
- Mixed-precision training fundamentals
- ZeRO (Zero Redundancy Optimizer) and FSDP sharding strategies
Framed by the '120 GB model won't fit on a 32 GB GPU' problem, it's a solid resource for engineers learning distributed training memory optimization.
More from Infra
- 303 companies mapped on Nebius: from 1X robots to Mastercard, tracked in independent atlas — pdamodaran · 2026-10-19
- vLLM details MiniMax M3.1 Flash serving stack: 1.7x faster indexer decode via NVFP4 — vllm_project · 2026-10-19
- Self-hosted DeepSeek inference beats every vendor on price/perf, boosting the open-weight enterprise case — bendee983 · 2026-10-19
- DeepSeek V4.1 Flash at 2.61 bpw hits 26 tok/s on dual RTX 3090s with dsx — whiteh4cker · 2026-10-19
- Neocloud shakeout ahead: only 7-10 of hundreds of AI clouds may survive, Nvidia most exposed — The Decoder · 2026-10-19
- Wan 2.2 I2V on a 12GB GPU uses only 8-9GB VRAM—system RAM is the real bottleneck — noise_to_motion · 2026-10-19