Understand onnxruntime.SessionOptions() intra_op_num_threads – Onnx Tutorial

By | October 8, 2023

In this tutorial, we will introduce onnxruntime.SessionOptions() intra_op_num_threads.

For example:

import onnxruntime as rt

sess_options = rt.SessionOptions()

sess_options.intra_op_num_threads = 0
sess_options.execution_mode = rt.ExecutionMode.ORT_SEQUENTIAL
sess_options.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_ALL
sess_options.add_session_config_entry("session.intra_op.allow_spinning", "1")

intra_op_num_threads: it controls the total number of INTRA threads to use to run the model.

Default: (not specified or 0). sess_options.intra_op_num_threads = 0

INTRA Threads Total = Number of physical CPU Cores. Leaving at default also enables some affinitization.

E.g. 6-core machine (with 12 HT logical processors) = 6 total INTRA threads

You can learn more by this tutorial:


When sess_options.intra_op_num_threads = 1

If you plan to test the performance of onnx, you can set sess_options.intra_op_num_threads = 1. For example, if you plan to compute rtf metric.