One diagram explains data vs tensor vs pipeline parallelism, plus UvA tutorial

SeunghyunSEO7 · x · 2026-11-19

When asked to explain tensor parallelism, the reply cites UvA DL Notebooks' tutorial with a concise explanation: data parallelism splits the batch across GPUs each holding full weights, while tensor parallelism keeps the same batch but splits weights along feature dimensions across GPUs, then merges partial computations. The shared diagram compares single GPU, DP, TP, and pipeline parallelism side by side; the full tutorial notebooks are linked.

Related event: Explaining DP, TP and PP Parallelism in One Picture(2 posts)→

Original post →

More from Research

Research channel →