Why might TensorRT FP16 inference be slower on a GPU without suitable native FP16 support?
GPU / CUDALevel: IntermediateHardware support for reduced precision
Why might TensorRT FP16 inference be slower on a GPU without suitable native FP16 support?
Hints
- Storage precision and native arithmetic throughput differ.
- What conversions or kernel choices appear in the execution plan?
Answer
Reducing storage precision does not guarantee faster arithmetic or a better kernel. Hardware support, chosen tactics, conversions and memory traffic determine performance. Inspect warnings and the precision actually used by layers, then benchmark equivalent workloads and numerical accuracy against FP32. Do not infer expected throughput solely from the fact that FP16 uses fewer bytes.
Source and licence: OpenMMLab — MMDeploy FAQ · Apache-2.0 · answer: Quantoros editorial