How do you separate slow compilation from slow execution in a torch.compile benchmark?
GPU / CUDALevel: IntermediateCompilation latency diagnosis
How do you separate slow compilation from slow execution in a torch.compile benchmark?
Hints
- Do first call and warmed calls perform the same work?
- Track recompilations and cache state explicitly.
Answer
Measure first-call compilation, any later recompilations and warmed execution separately. Use compiler timing/debug traces to locate expensive phases; keep shape coverage and cache state explicit. Warm-up should resemble production inputs. Report cold-start latency as well as steady-state speed, because a short-lived service may never amortize compilation cost.
Source and licence: PyTorch contributors — torch.compile FAQ · BSD-3-Clause · answer: Quantoros editorial