Why can fusing consecutive tensor operations improve performance without reducing their arithmetic?
GPU / CUDALevel: IntermediateKernel fusion and memory traffic
Why can fusing consecutive tensor operations improve performance without reducing their arithmetic?
Hints
- Count intermediate memory writes and reads.
- What new pressure can a larger fused kernel create?
Answer
Fusion can avoid writing intermediates to device memory and reading them again, and reduce kernel-launch overhead. The gain depends on memory traffic, register pressure, occupancy and tensor sizes, not just FLOP count. Inductor may generate Triton kernels for suitable GPU work; that Triton language is distinct from NVIDIA Triton Inference Server. Benchmark the fused path rather than assuming it always wins.
Source and licence: PyTorch contributors — torch.compile FAQ · BSD-3-Clause · answer: Quantoros editorial