PP vs DP vs TP explained: how weights and batches split across GPUs, and where bubbles come from
SeunghyunSEO7 · x · 2026-11-19
- Replying to @yacineMTB on inference backend parallelism, the author explains three common strategies:
- PP (pipeline parallelism): weights are split across depth; gpu0 processes a batch and P2P-sends activations to gpu1 — same as pipelining in other backend jobs, with idle bubbles while GPUs wait.
- DP (data parallelism): the batch is split, each GPU processes a different mini-batch with the full weights.
- TP (tensor parallelism): every GPU sees the same batch but holds partial weights, does a partial operation, then merges results across GPUs.
Related event: Explaining DP, TP and PP Parallelism in One Picture(2 posts)→
More from Infra
- Late Entry in looped LLMs delivers up to 12.5x lossy and 8.4x exact speedups — NanyangTechnologicalUniversity · 2026-11-19
- CoreWeave Insolvency Speculation: $24.6B Debt Due vs $3.7B Quarterly Revenue — SumitGup · 2026-11-19
- Anthropic priced its IPO at $83 above range; anon thread predicts the stock halves before unlocks — basedjensen · 2026-11-19
- Satya Nadella says Microsoft would lease SpaceX compute for data centers in space — elonmusk · 2026-11-19
- AMD ROCm 10.2 Ships With 18 New Ryzen Laptop Models Supported for Local AI — AnushElangovan · 2026-11-19
- Consumer NAND spot prices drop 20% in a month, more SSD relief expected in early 2027 — SumitGup · 2026-11-19