Deploy machine learned models with ONNX

Links: notebook, html, python, slides, GitHub

Xavier Dupré - Senior Data Scientist at Microsoft - Computer Science Teacher at ENSAE

Most of machine learning libraries are optimized to train models and not necessarily to use them for fast predictions in online web services. ONNX is one solution started last year by Microsoft and Facebook. This presentation describes the concept and shows some examples with scikit-learn and ML.net.

GitHub repos

Contributing to

from jyquickhelper import add_notebook_menu
add_notebook_menu(last_level=2)
%matplotlib inline
import matplotlib.pyplot as plt
from pyquickhelper.helpgen import NbImage

Open source tools in this talk

import keras, lightgbm, onnx, skl2onnx, onnxruntime, sklearn, torch, xgboost
mods = [keras, lightgbm, onnx, skl2onnx, onnxruntime, sklearn, torch, xgboost]
for m in mods:
    print(m.__name__, m.__version__)
Using TensorFlow backend.
keras 2.2.4
lightgbm 2.2.2
onnx 1.4.1
skl2onnx 1.4.3
onnxruntime 0.3.0
sklearn 0.21.dev0
torch 1.0.1
xgboost 0.81

Learn and predict

  • Two different purposes not necessarily aligned for optimization

  • Learn : computation optimized for large number of observations (batch prediction)

  • Predict : computation optimized for one observation (one-off prediction)

  • Machine learning libraries optimize the learn scenario.

One-off prediction with random forests

Benchmark of libraries for a regression problem.

from sklearn.datasets import load_diabetes
diabetes = load_diabetes()
diabetes_X_train = diabetes.data[:-20]
diabetes_X_test  = diabetes.data[-20:]
diabetes_y_train = diabetes.target[:-20]
diabetes_y_test  = diabetes.target[-20:]
diabetes_X_train[:1]
array([[ 0.03807591,  0.05068012,  0.06169621,  0.02187235, -0.0442235 ,
        -0.03482076, -0.04340085, -0.00259226,  0.01990842, -0.01764613]])
from jupytalk.benchmark import make_dataframe
df = make_dataframe(diabetes_y_train, diabetes_X_train)
df.to_csv("diabetes.csv", index=False)
df.head(n=2)
Label F0 F1 F2 F3 F4 F5 F6 F7 F8 F9
0 151.0 0.038076 0.050680 0.061696 0.021872 -0.044223 -0.034821 -0.043401 -0.002592 0.019908 -0.017646
1 75.0 -0.001882 -0.044642 -0.051474 -0.026328 -0.008449 -0.019163 0.074412 -0.039493 -0.068330 -0.092204
from jupytalk.benchmark import timeexec
measures_rf = []
from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor(n_estimators=10)
rf.fit(diabetes_X_train, diabetes_y_train)
RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
                      max_features='auto', max_leaf_nodes=None,
                      min_impurity_decrease=0.0, min_impurity_split=None,
                      min_samples_leaf=1, min_samples_split=2,
                      min_weight_fraction_leaf=0.0, n_estimators=10,
                      n_jobs=None, oob_score=False, random_state=None,
                      verbose=0, warm_start=False)
measures_rf += [timeexec("sklearn", "rf.predict(diabetes_X_test[:1])",
                         context=globals())]
Average: 922.53 µs deviation 365.73 µs (with 50 runs) in [571.02 µs, 1.49 ms]
from xgboost import XGBRegressor
xg = XGBRegressor(n_estimators=10)
xg.fit(diabetes_X_train, diabetes_y_train)
XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,
             colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0,
             max_depth=3, min_child_weight=1, missing=None, n_estimators=10,
             n_jobs=1, nthread=None, objective='reg:linear', random_state=0,
             reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
             silent=True, subsample=1)
measures_rf += [timeexec("xgboost", "xg.predict(diabetes_X_test[:1])",
                         context=globals())]
Average: 74.00 µs deviation 15.12 µs (with 50 runs) in [63.13 µs, 105.74 µs]
from lightgbm import LGBMRegressor
lg = LGBMRegressor(n_estimators=10)
lg.fit(diabetes_X_train, diabetes_y_train)
LGBMRegressor(boosting_type='gbdt', class_weight=None, colsample_bytree=1.0,
              importance_type='split', learning_rate=0.1, max_depth=-1,
              min_child_samples=20, min_child_weight=0.001, min_split_gain=0.0,
              n_estimators=10, n_jobs=-1, num_leaves=31, objective=None,
              random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=True,
              subsample=1.0, subsample_for_bin=200000, subsample_freq=0)
measures_rf += [timeexec("lightgbm", "lg.predict(diabetes_X_test[:1])",
                         context=globals())]
Average: 144.67 µs deviation 43.91 µs (with 50 runs) in [100.93 µs, 225.89 µs]

This would require to reimplement the prediction function.

import pandas
df = pandas.DataFrame(data=measures_rf)
df = df.set_index("legend").sort_values("average")
df
average code deviation first first3 last3 max5 min5 repeat run
legend
xgboost 0.000074 xg.predict(diabetes_X_test[:1]) 0.000015 0.000116 0.000117 0.000069 0.000106 0.000063 200 50
lightgbm 0.000145 lg.predict(diabetes_X_test[:1]) 0.000044 0.000191 0.000185 0.000121 0.000226 0.000101 200 50
sklearn 0.000923 rf.predict(diabetes_X_test[:1]) 0.000366 0.000833 0.000699 0.000643 0.001492 0.000571 200 50
%matplotlib inline
import matplotlib.pyplot as plt
fig, ax = plt.subplots(1, 1, figsize=(10,3))
df[["average", "deviation"]].plot(kind="barh", logx=True, ax=ax, xerr="deviation",
                                  legend=False, fontsize=12, width=0.8)
ax.set_ylabel("")
ax.grid(b=True, which="major")
ax.grid(b=True, which="minor")
ax.set_title("Prediction time for one observation\nRandom Forest (10 trees)");
../_images/onnx_deploy_pyparis_24_0.png

Keep in mind

  • Trained trees are not necessarily the same.

  • Performance is not compared.

  • Order of magnitude is important here.

What is batch prediction?

  • Instead of running N times 1 prediction

  • We run 1 time N predictions

The code can be found at MS Experience 2018.

NbImage('batch.png', width=600)
../_images/onnx_deploy_pyparis_27_0.png

ONNX

ONNX can represent any pipeline of data.

Let’s visualize a machine learning pipeline (see the code at MS Experience).

NbImage("pipeviz.png", width=500)
../_images/onnx_deploy_pyparis_29_0.png

ONNX = language to describe models

  • Standard format to describe machine learning

  • Easier to exchange, export

ONNX = machine learning oriented

Can represent any mathematical function handling numerical and text features.

NbImage("onnxop.png", width=600)
../_images/onnx_deploy_pyparis_32_0.png

actively supported

  • Microsoft

  • Facebook

  • first created to deploy deep learning models

  • extended to other models

Train somewhere, predict somewhere else

Cannot optimize the code for both training and predicting.

Training

Predicting

Batch prediction

One-off prediction

Huge memory

Small memory

Huge data

Small data

.

High latency

Libraries for predictions

  • Optimized for predictions

  • Optimized for a device

ONNX Runtime

ONNX Runtime for inferencing machine learning models now in preview

Dedicated runtime for:

  • CPU

  • GPU

ONNX on random forest

NbImage("process.png", width=500)
../_images/onnx_deploy_pyparis_39_0.png
rf
RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
                      max_features='auto', max_leaf_nodes=None,
                      min_impurity_decrease=0.0, min_impurity_split=None,
                      min_samples_leaf=1, min_samples_split=2,
                      min_weight_fraction_leaf=0.0, n_estimators=10,
                      n_jobs=None, oob_score=False, random_state=None,
                      verbose=0, warm_start=False)

Conversion to ONNX

sklearn-onnx

from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType
model_onnx = convert_sklearn(rf, "rf_diabetes",
                             [('input', FloatTensorType([1, 10]))])
The maximum opset needed by this model is only 1.
print(str(model_onnx)[:450] + "\n...")
ir_version: 4
producer_name: "skl2onnx"
producer_version: "1.4.3"
domain: "ai.onnx"
model_version: 0
doc_string: ""
graph {
  node {
    input: "input"
    output: "variable"
    name: "TreeEnsembleRegressor"
    op_type: "TreeEnsembleRegressor"
    attribute {
      name: "n_targets"
      i: 1
      type: INT
    }
    attribute {
      name: "nodes_falsenodeids"
      ints: 334
      ints: 235
      ints: 234
      ints: 119
      ints: 86
...

Save the model

with open('rf_sklearn.onnx', "wb") as f:
    f.write(model_onnx.SerializeToString())

Compute predictions

import onnxruntime

sess = onnxruntime.InferenceSession("rf_sklearn.onnx")

for i in sess.get_inputs():
    print('Input:', i)
for o in sess.get_outputs():
    print('Output:', o)
Input: NodeArg(name='input', type='tensor(float)', shape=[1, 10])
Output: NodeArg(name='variable', type='tensor(float)', shape=[1, 1])
import numpy

def predict_onnxrt(x):
    return sess.run(["variable"], {'input': x})

print("Prediction:", predict_onnxrt(diabetes_X_test[:1].astype(numpy.float32)))
Prediction: [array([[195.8]], dtype=float32)]
measures_rf += [timeexec("onnx", "predict_onnxrt(diabetes_X_test[:1].astype(numpy.float32))",
                         context=globals())]
Average: 20.91 µs deviation 5.96 µs (with 50 runs) in [15.12 µs, 30.55 µs]
fig, ax = plt.subplots(1, 1, figsize=(10,3))
df = pandas.DataFrame(data=measures_rf)
df = df.set_index("legend").sort_values("average")
df[["average", "deviation"]].plot(kind="barh", logx=True, ax=ax, xerr="deviation",
                                  legend=False, fontsize=12, width=0.8)
ax.set_ylabel("")
ax.grid(b=True, which="major")
ax.grid(b=True, which="minor")
ax.set_title("Prediction time for one observation\nRandom Forest (10 trees)");
../_images/onnx_deploy_pyparis_50_0.png

Deep learning

  • transfer learning with keras

  • orther convert pytorch, caffee…

Code is available at MS Experience 2018.

Perf

NbImage("dlpref.png", width=600)
../_images/onnx_deploy_pyparis_53_0.png

Model zoo

Converted Models

NbImage("zoo.png", width=800)
../_images/onnx_deploy_pyparis_55_0.png

Tiny yolo

Source: TinyYOLOv2 on onnx

from pyensae.datasource import download_data
download_data("tiny_yolov2.tar.gz",
              url="https://onnxzoo.blob.core.windows.net/models/opset_8/tiny_yolov2/")
['.\tiny_yolov2/Model.onnx',
 '.\tiny_yolov2/test_data_set_0/input_0.pb',
 '.\tiny_yolov2/test_data_set_0/output_0.pb',
 '.\tiny_yolov2/test_data_set_1/input_0.pb',
 '.\tiny_yolov2/test_data_set_1/output_0.pb',
 '.\tiny_yolov2/test_data_set_2/input_0.pb',
 '.\tiny_yolov2/test_data_set_2/output_0.pb']
sess = onnxruntime.InferenceSession("tiny_yolov2/Model.onnx")
for i in sess.get_inputs():
    print('Input:', i)
for o in sess.get_outputs():
    print('Output:', o)
Input: NodeArg(name='image', type='tensor(float)', shape=[None, 3, 416, 416])
Output: NodeArg(name='grid', type='tensor(float)', shape=[None, 125, 13, 13])
from PIL import Image,ImageDraw
img = Image.open('Au-Salon-de-l-agriculture-la-campagne-recrute.jpg')
img
../_images/onnx_deploy_pyparis_59_0.png
img2 = img.resize((416, 416))
img2
../_images/onnx_deploy_pyparis_60_0.png
X = numpy.asarray(img2)
X = X.transpose(2,0,1)
X = X.reshape(1,3,416,416)

out = sess.run(None, {'image': X.astype(numpy.float32)})
out = out[0][0]
def display_yolo(img, seuil):
    import numpy as np
    numClasses = 20
    anchors = [1.08, 1.19, 3.42, 4.41, 6.63, 11.38, 9.42, 5.11, 16.62, 10.52]

    def sigmoid(x, derivative=False):
        return x*(1-x) if derivative else 1/(1+np.exp(-x))

    def softmax(x):
        scoreMatExp = np.exp(np.asarray(x))
        return scoreMatExp / scoreMatExp.sum(0)

    clut = [(0,0,0),(255,0,0),(255,0,255),(0,0,255),(0,255,0),(0,255,128),
            (128,255,0),(128,128,0),(0,128,255),(128,0,128),
            (255,0,128),(128,0,255),(255,128,128),(128,255,128),(255,255,0),
            (255,128,128),(128,128,255),(255,128,128),(128,255,128),(128,255,128)]
    label = ["aeroplane","bicycle","bird","boat","bottle",
             "bus","car","cat","chair","cow","diningtable",
             "dog","horse","motorbike","person","pottedplant",
             "sheep","sofa","train","tvmonitor"]

    draw = ImageDraw.Draw(img)
    for cy in range(0,13):
        for cx in range(0,13):
            for b in range(0,5):
                channel = b*(numClasses+5)
                tx = out[channel  ][cy][cx]
                ty = out[channel+1][cy][cx]
                tw = out[channel+2][cy][cx]
                th = out[channel+3][cy][cx]
                tc = out[channel+4][cy][cx]

                x = (float(cx) + sigmoid(tx))*32
                y = (float(cy) + sigmoid(ty))*32

                w = np.exp(tw) * 32 * anchors[2*b  ]
                h = np.exp(th) * 32 * anchors[2*b+1]

                confidence = sigmoid(tc)

                classes = np.zeros(numClasses)
                for c in range(0,numClasses):
                    classes[c] = out[channel + 5 +c][cy][cx]
                    classes = softmax(classes)
                detectedClass = classes.argmax()

                if seuil < classes[detectedClass]*confidence:
                    color =clut[detectedClass]
                    x = x - w/2
                    y = y - h/2
                    draw.line((x  ,y  ,x+w,y ),fill=color, width=3)
                    draw.line((x  ,y  ,x  ,y+h),fill=color, width=3)
                    draw.line((x+w,y  ,x+w,y+h),fill=color, width=3)
                    draw.line((x  ,y+h,x+w,y+h),fill=color, width=3)

    return img
img2 = img.resize((416, 416))
display_yolo(img2, 0.038)
../_images/onnx_deploy_pyparis_63_0.png

Conclusion

  • ONNX is a working progress, active development

  • ONNX is open source

  • ONNX does not depend on the machine learning framework

  • ONNX provides dedicated runtimes

  • ONNX is fast and available in Python…

Metadata to trace deployed models

meta = sess.get_modelmeta()
meta.description
"The Tiny YOLO network from the paper 'YOLO9000: Better, Faster, Stronger' (2016), arXiv:1612.08242"
meta.producer_name, meta.version
('WinMLTools', 0)