What is tensor parallelism vs pipeline parallelism?
ML System DesignLevel: IntermediateParallelism Strategies
What is tensor parallelism vs pipeline parallelism?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
When a model doesn't fit on one GPU, distribute it across multiple GPUs:
Tensor Parallelism (TP): split individual weight matrices across GPUs.
Weight matrix W (d×d) split across 4 GPUs:
GPU0: W[:, :d/4] GPU1: W[:, d/4:d/2]
GPU2: W[:, d/2:3d/4] GPU3: W[:, 3d/4:]
- All GPUs active during each forward pass layer
- Requires high-bandwidth interconnect (NVLink) between GPUs
- Best within a single node (NVLink bandwidth >> PCIe or InfiniBand)
Pipeline Parallelism (PP): split model layers across GPUs.
GPU0: Layers 1-20 GPU1: Layers 21-40
GPU2: Layers 41-60 GPU3: Layers 61-80
- Micro-batch pipelining to keep all GPUs busy
- Works across nodes (lower bandwidth interconnect OK)
- Pipeline bubbles reduce efficiency at small batch sizes
In practice: use TP within a node, PP across nodes. 70B model on 2 nodes × 8 A100s = TP=8 + PP=2.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0