Skip to content
Quantoros

How do you separate slow compilation from slow execution in a torch.compile benchmark?

GPU / CUDALevel: IntermediateCompilation latency diagnosis

How do you separate slow compilation from slow execution in a torch.compile benchmark?

Hints
  • Do first call and warmed calls perform the same work?
  • Track recompilations and cache state explicitly.

Answer

Measure first-call compilation, any later recompilations and warmed execution separately. Use compiler timing/debug traces to locate expensive phases; keep shape coverage and cache state explicit. Warm-up should resemble production inputs. Report cold-start latency as well as steady-state speed, because a short-lived service may never amortize compilation cost.

Source and licence: PyTorch contributors — torch.compile FAQ · BSD-3-Clause · answer: Quantoros editorial

Related questions