Benchmark (ONNX) for common datasets (classification)#

Overview#

The following graph plots the ratio between onnxruntime and scikit-learn. It looks into multiple models for a couple of datasets : breast cancer, digits. It computes the prediction time for the following models:

  • ADA: AdaBoostClassifier()

  • BNB: BernoulliNB()

  • DT: DecisionTreeClassifier(max_depth=6)

  • GBT: GradientBoostingClassifier(max_depth=6, n_estimators=100)

  • KNN: KNeighborsClassifier()

  • KNN-cdist: KNeighborsClassifier(), the conversion to ONNX is run with option {'optim': 'cdist'} to use a specific operator to compute pairwise distances

  • LGB: LGBMClassifier(max_depth=6, n_estimators=100)

  • LR: LogisticRegression(solver="liblinear", penalty="l2")

  • LR-ZM: LogisticRegression(solver="liblinear", penalty="l2"), the conversion to ONNX is run with option {'zipmap': False} to remove the final zipmap operator.

  • MLP: MLPClassifier()

  • MNB: MultinomialNB()

  • NuSVC: NuSVC(probability=True)

  • OVR: OneVsRestClassifier(DecisionTreeClassifier(max_depth=6))

  • RF: RandomForestClassifier(max_depth=6, n_estimators=100)

  • SVC: SVC(probability=True)

  • XGB: XGBClassifier(max_depth=6, n_estimators=100)

The predictor follows a StandardScaler (or a MinMaxScaler if the model is a Naive Bayes one) in a pipeline if norm=True or is the only object is norm=False. The pipeline looks like make_pipeline(StandardScaler(), estimator()). Three runtimes are tested:

  • skl: scikit-learn,

  • ort: onnxruntime,

  • pyrt: mlprodict, it relies on numpy for most of the operators except trees and svm which use a modified version of the C++ code embedded in onnxruntime,

  • pyrtc: same runtime as pyrt but the graph logic is replaced by a function dynamically compiled when the ONNX file is loaded.

(Source code, png, hires.png, pdf)

../_images/onnxruntime_datasets_num-1.png

Graph X = number of observations to predict#

(Source code, png, hires.png, pdf)

../_images/onnxruntime_datasets_num-2.png

Graph computing time per observations#

The following graph shows the computing cost per observations depending on the batch size. scikit-learn is clearly optimized for batch predictions (= training).

(Source code, png, hires.png, pdf)

../_images/onnxruntime_datasets_num-3.png

Graph of differences between scikit-learn and onnxruntime#

(Source code, png, hires.png, pdf)

../_images/onnxruntime_datasets_num-4.png

Graph of differences between scikit-learn and python runtime#

(Source code, png, hires.png, pdf)

../_images/onnxruntime_datasets_num-5.png

Configuration#

<<<

from pyquickhelper.pandashelper import df2rst
import pandas
name = os.path.join(
    __WD__, "../../onnx/results/bench_plot_datasets_num.time.csv")
df = pandas.read_csv(name)
print(df2rst(df, number_format=4))

>>>

name

version

value

date

2020-01-29

python

3.7.2 (default, Mar 1 2019, 18:34:21) [GCC 6.3.0 20170516]

platform

linux

OS

Linux-4.9.0-8-amd64-x86_64-with-debian-9.6

machine

x86_64

processor

release

4.9.0-8-amd64

architecture

(β€˜64bit’, β€˜β€™)

mlprodict

0.3

numpy

1.18.1

openblas, language=c

onnx

1.6.37

opset=12

onnxruntime

1.1.995

CPU-DNNL-MKL-ML

pandas

0.25.3

skl2onnx

1.6.994

sklearn

0.22.1

Raw results#

bench_plot_datasets_num.csv

<<<

from pyquickhelper.pandashelper import df2rst
from pymlbenchmark.benchmark.bench_helper import bench_pivot
import pandas
name = os.path.join(
    __WD__, "../../onnx/results/bench_plot_datasets_num.perf.csv")
df = pandas.read_csv(name)
print(df2rst(df, number_format=4))