Skip to content
Quantoros

What is tensor parallelism vs pipeline parallelism?

ML System DesignLevel: IntermediateParallelism Strategies

What is tensor parallelism vs pipeline parallelism?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

When a model doesn't fit on one GPU, distribute it across multiple GPUs:

Tensor Parallelism (TP): split individual weight matrices across GPUs.

Weight matrix W (d×d) split across 4 GPUs:
GPU0: W[:, :d/4]   GPU1: W[:, d/4:d/2]
GPU2: W[:, d/2:3d/4]   GPU3: W[:, 3d/4:]
  • All GPUs active during each forward pass layer
  • Requires high-bandwidth interconnect (NVLink) between GPUs
  • Best within a single node (NVLink bandwidth >> PCIe or InfiniBand)

Pipeline Parallelism (PP): split model layers across GPUs.

GPU0: Layers 1-20   GPU1: Layers 21-40
GPU2: Layers 41-60   GPU3: Layers 61-80
  • Micro-batch pipelining to keep all GPUs busy
  • Works across nodes (lower bandwidth interconnect OK)
  • Pipeline bubbles reduce efficiency at small batch sizes

In practice: use TP within a node, PP across nodes. 70B model on 2 nodes × 8 A100s = TP=8 + PP=2.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions