A 10-item beginner reading list for LLM distributed training, from Horovod to FSDP
East-Muffin-6472 · reddit · 2026-11-17
The author shares a 10-item reading list in the exact order they learned LLM distributed training from scratch, aimed at engineers on small multi-node clusters:
- Umar Jamil's first-principles distributed training framework video
- Horovod: Fast and Easy Distributed Deep Learning in TensorFlow
- PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Large Scale Distributed Deep Networks (DistBelief)
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Megatron-LM: model parallelism for multi-billion parameter LMs
- Efficient Large-Scale LLM Training on GPU Clusters Using Megatron-LM
- GPipe: pipeline parallelism for giant networks
- ZeRO-Infinity: breaking the GPU memory wall
- PyTorch FSDP: fully sharded data parallel
Hosted as a curated shelf on papercase.dev with community-driven rebalancing of the reading order.
More from Infra
- 18MB browser-run TTS model tiny-voca makes paid English TTS look silly — thisdudelikesAI · 2026-11-17
- Replicate quietly switches nano banana outputs from JPG downloads to presigned URLs — Kyrannio · 2026-11-17
- Fireworks AI tells investors chips are a commodity, its moat is software that runs models fast — hargup13 · 2026-11-17
- Claude-written CUDA C++ port runs TRELLIS.2 5.3x faster on H200, bit-identical — huggingface · 2026-11-17
- China to make 3M AI ASICs in 2027 vs West's 20M, each ~4x faster per chip — teortaxesTex · 2026-11-17
- Harmonic's AI theorem prover Aristotle takes a no-hallucination hardware synthesis talk to Cambridge — satnam6502 · 2026-11-17