How do you constrain ONNX Runtime CPU inference to avoid thread oversubscription?
ML System DesignLevel: IntermediateCPU inference thread budgets
How do you constrain ONNX Runtime CPU inference to avoid thread oversubscription?
Hints
- Distinguish parallel operators from parallel work inside one operator.
- Count concurrent sessions and external library threads too.
Answer
Distinguish intra-op parallelism within an operator from inter-op scheduling between operators, and account for the build's thread runtime. For a single-thread test, set the relevant intra-op thread count to one and use sequential execution; OpenMP builds may also require an OMP_NUM_THREADS limit. Budget threads across concurrent sessions and external libraries, then measure throughput and tail latency.
Source and licence: Microsoft — ONNX Runtime FAQ · MIT · answer: Quantoros editorial