Skip to content
Quantoros

Why can fusing consecutive tensor operations improve performance without reducing their arithmetic?

GPU / CUDALevel: IntermediateKernel fusion and memory traffic

Why can fusing consecutive tensor operations improve performance without reducing their arithmetic?

Hints
  • Count intermediate memory writes and reads.
  • What new pressure can a larger fused kernel create?

Answer

Fusion can avoid writing intermediates to device memory and reading them again, and reduce kernel-launch overhead. The gain depends on memory traffic, register pressure, occupancy and tensor sizes, not just FLOP count. Inductor may generate Triton kernels for suitable GPU work; that Triton language is distinct from NVIDIA Triton Inference Server. Benchmark the fused path rather than assuming it always wins.

Source and licence: PyTorch contributors — torch.compile FAQ · BSD-3-Clause · answer: Quantoros editorial

Related questions