Benchmarks#

Validates a runtime against scikit-learn#

This reuse the example Measure ONNX runtime performances. The goal is to compare different implementation of an operator.

Step 1: create a virtual environment

python -m virtualenv baseline

All other steps are executed within the local environment.

Step 2: install the packages

pip install numpy scikit-learn onnx onnxruntime skl2onnx pyquickhelper matplotlib mlprodict threadpoolctl lightgbm xgboost

Step 3: run the benchmark

export model="RandomForestRegressor"
python -m mlprodict validate_runtime --n_features 4,50 -nu 2 -re 2 -o 11 -op 11 -v 1 --out_raw data$model.csv --out_summary summary$model.csv -b 1 --dump_folder dump_errors --runtime python_compiled,onnxruntime1 --models $model --out_graph bench_png$model --dtype 32

A full example is available on the following page.

Compares two different onnxruntime#

The benchmark is done using asv. This section assumes there exist a local pypi server with the different tested versions. It can be done with module pypiserver.

Step 0: pypiserver

Copy all the necessary packages in a local folder and starts a local pypiserver.

python -m pypiserver -u -p 8067 --disable-fallback local_pypi_server

Step 1: virtual environment

python -m virtualenv pyenv

Everything else is done from the virtual environment.

pip install numpy scikit-learn matplotlib pandas cython pybind11 lightgbm xgboost virtualenv pyquickhelper

The module asv may work, otherwise a modified version is necessary:

pip install git+https://github.com/sdpython/asv.git@jenkins

And the local packages:

pip install --upgrade --index http://localhost:8067/simple/ skl2onnx onnxconverter_common mlprodict onnxruntime onnx --extra-index-url=https://pypi.python.org/simple/

Step 2: create the benchmark

Versions must be verified.

python -m mlprodict asv_bench  --location . --models "RandomForestRegressor" --build "build" -dt 32 -ma "{\"onnxruntime\":[\"1.1.2\", \"http://localhost:8067/simple/\"],\"onnx\":[\"1.6.0\"],\"scikit-learn\":[\"0.22.2.post1\"]}" -n 4 -o 11 -op 11 -r scikit-learn,python_compiled,onnxruntime1

Step 3: run the benchmak

python -m asv run --show-stderr --config=asv.conf.json

Step 4: publish the bechmark

python -m asv publish -o html --config=asv.conf.json

Step 5: export into ASV format

python -m mlprodict asv2csv -f . -o asv_benchmark.csv -b skl -c asv.conf.json