Skip to content
Quantoros

How do you constrain ONNX Runtime CPU inference to avoid thread oversubscription?

ML System DesignLevel: IntermediateCPU inference thread budgets

How do you constrain ONNX Runtime CPU inference to avoid thread oversubscription?

Hints
  • Distinguish parallel operators from parallel work inside one operator.
  • Count concurrent sessions and external library threads too.

Answer

Distinguish intra-op parallelism within an operator from inter-op scheduling between operators, and account for the build's thread runtime. For a single-thread test, set the relevant intra-op thread count to one and use sequential execution; OpenMP builds may also require an OMP_NUM_THREADS limit. Budget threads across concurrent sessions and external libraries, then measure throughput and tail latency.

Source and licence: Microsoft — ONNX Runtime FAQ · MIT · answer: Quantoros editorial

Related questions