Skip to content
Quantoros

Why can compiling a DDP backward pass affect communication-computation overlap?

GPU / CUDALevel: AdvancedCompilation and communication overlap

Why can compiling a DDP backward pass affect communication-computation overlap?

Hints
  • When does DDP know that a gradient bucket is ready?
  • A larger compiled graph can change hook timing.

Answer

This answer is part of mentoring

Advanced questions come with a full breakdown and a mock interview with a working quant researcher. The first call is free.

Source and licence: PyTorch contributors — torch.compile FAQ · BSD-3-Clause · answer: Quantoros editorial

Related questions