Measure ONNX runtime performances

The following example shows how to use the command line to compare one or two runtimes with scikit-learn. It relies on function validate_runtime which can be called from python or through a command line described in page Command lines.

Run the benchmark

The following line creates a folder used to dump information about models which failed during the benchmark.

import os
if not os.path.exists("dump_errors"):
    os.mkdir("dump_errors")

The benchmark can be run with a python instruction or a command line:

python -m mlprodict validate_runtime -v 1 --out_raw data.csv --out_summary summary.csv
           -b 1 --dump_folder dump_errors --runtime python,onnxruntime1
           --models LinearRegression,DecisionTreeRegressor
           --n_features 4,10 --out_graph bench_png
           -t "{\"1\":{\"number\":10,\"repeat\":10},\"10\":{\"number\":5,\"repeat\":5}}"

We use the python instruction in this example.

from mlprodict.cli import validate_runtime

validate_runtime(
    verbose=1,
    out_raw="data.csv", out_summary="summary.csv",
    benchmark=True, dump_folder="dump_errors",
    runtime=['python', 'onnxruntime1'],
    models=['LinearRegression', 'DecisionTreeRegressor'],
    n_features=[4, 10], dtype="32",
    out_graph="bench.png",
    time_kwargs={
        1: {"number": 100, "repeat": 100},
        10: {"number": 50, "repeat": 50},
        100: {"number": 40, "repeat": 50},
        1000: {"number": 40, "repeat": 40},
        10000: {"number": 20, "repeat": 20},
    }
)
RT/SKL-N=1, N=10, N=100, N=1000, N=10000

Out:

time_kwargs={1: {'number': 100, 'repeat': 100}, 10: {'number': 50, 'repeat': 50}, 100: {'number': 40, 'repeat': 50}, 1000: {'number': 40, 'repeat': 40}, 10000: {'number': 20, 'repeat': 20}}
[enumerate_validated_operator_opsets] opset in [12, None].

  0%|          | 0/2 [00:00<?, ?it/s]
LinearRegression            :   0%|          | 0/2 [00:00<?, ?it/s][enumerate_compatible_opset] opset in [12, None].

LinearRegression            :  50%|#####     | 1/2 [00:53<00:53, 53.10s/it]
DecisionTreeRegressor       :  50%|#####     | 1/2 [00:53<00:53, 53.10s/it][enumerate_compatible_opset] opset in [12, None].

DecisionTreeRegressor       : 100%|##########| 2/2 [01:56<00:00, 56.08s/it]
DecisionTreeRegressor       : 100%|##########| 2/2 [01:56<00:00, 58.07s/it]
Saving raw_data into 'data.csv'.
Saving summary into 'summary.csv'.
Saving graph into 'bench.png'.

Let’s show the results.

import pandas
df = pandas.read_csv("summary.csv")
df.head(n=2).T
0 1
name DecisionTreeRegressor DecisionTreeRegressor
problem b-reg b-reg
scenario default default
optim NaN NaN
method_name predict predict
output_index 0 0
conv_options {} {}
inst {"random_state": 42} {"random_state": 42}
n_features 4 4
runtime onnxruntime1 python
skl_version 0.23.2 0.23.2
skl_nop 1 1
skl_ncoef -1 -1
skl_nlin -1 -1
skl_nnodes 223 223
skl_ntrees 1 1
skl_max_depth 13 13
onx_size 8939 8939
onx_nnodes 1 1
onx_ninits 0 0
onx_producer_name skl2onnx skl2onnx
onx_producer_version 1.7.1092 1.7.1092
onx_ai.onnx.ml 1 1
onx_size_optim 8939 8939
onx_nnodes_optim 1 1
onx_ninits_optim 0 0
opset12 OK /1 OK /1
RT/SKL-N=1 0.348199 0.216102
N=10 0.359123 0.220242
N=100 13.5856 47.6277
N=1000 5.75636 1.14743
N=10000 3.01263 5.4129
RT/SKL-N=1-min 0.329416 0.204646
RT/SKL-N=1-max 0.385676 0.243954
N=10-min 0.34191 0.211893
N=10-max 0.386629 0.235561
N=100-min 0.404301 0.260772
N=100-max 57.0963 57.9516
N=1000-min 0.355126 0.23766
N=1000-max 37.5768 35.1042
N=10000-min 0.273219 0.214131
N=10000-max 9.43882 9.66182


Let’s display the graph generated by the function.

import matplotlib.pyplot as plt
import matplotlib.image as mpimg

img = mpimg.imread('bench.png')
fig = plt.imshow(img)
fig.axes.get_xaxis().set_visible(False)
fig.axes.get_yaxis().set_visible(False)
plt.show()
plot onnx benchmark

Total running time of the script: ( 1 minutes 57.957 seconds)

Gallery generated by Sphinx-Gallery