How distributed LLM training actually works: 6 parallelism strategies, comms and memory bottlenecks
techNmak · x · 2026-10-17
techNmak shares a comprehensive diagram of distributed LLM training:
- Six parallelism strategies: Data Parallelism (replicate model, sync gradients), Tensor Parallelism (split within layers), Pipeline Parallelism (partition layers, schedule microbatches), ZeRO/FSDP (shard optimizer states/gradients/params), Sequence/Context Parallelism, and Expert Parallelism for MoE routing.
- Communication layer: all-reduce, all-gather, reduce-scatter and all-to-all move different data; overhead depends on strategy, topology and tensor sizes.
- Memory constraint: under mixed-precision Adam, a 7B model needs 112 GB of unsharded model state alone (16 bytes/param).
- Measurement: pipeline bubbles, sync delays and activation memory limit throughput, so measure tokens/sec, step time, peak memory, MFU and scaling efficiency—not GPU count.
The diagram also covers gradient accumulation, pipeline schedules and bubble math, GPU mesh composition, bottleneck diagnosis and checkpointing.
More from Infra
- Gartner's Neocloud Call Confuses Financial Exposure With Architectural Reality — DavidLinthicum · 2026-10-17
- Reliance welcomes Starlink competition in India: 11,000+ satellites in orbit vs Jio's 2027 test launch — elonmusk · 2026-10-17
- Microsoft completes pre-training of MAI-Base-2: 2.8T-param MoE, 4-12x more FLOPs-efficient than DeepSeek-V4-Pro-Base — NandoDF · 2026-10-17
- Apple research: a 7B model with 8 retries matches a 32B model using 4x less memory — CatAstro_Piyush · 2026-10-17
- Can a ternarized DeepSeek run on a single RTX 3090? Open problem: agent accuracy — casper_hansen_ · 2026-10-17
- MOSS-Image gets INT8 quants and a community LoRA fixing SFT banding — tomByrer · 2026-10-17