.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/plot_bbegin_measure_time.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_auto_examples_plot_bbegin_measure_time.py>`
        to download the full example code or to run this example in your browser via Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_plot_bbegin_measure_time.py:


Benchmark ONNX conversion
=========================

.. index:: benchmark

Example :ref:`l-simple-deploy-1` converts a simple model.
This example takes a similar example but on random data
and compares the processing time required by each option
to compute predictions.

.. contents::
    :local:


Training a pipeline
+++++++++++++++++++

.. GENERATED FROM PYTHON SOURCE LINES 19-48

.. code-block:: default

    import numpy
    from pandas import DataFrame
    from tqdm import tqdm
    from sklearn import config_context
    from sklearn.datasets import make_regression
    from sklearn.ensemble import (
        GradientBoostingRegressor, RandomForestRegressor,
        VotingRegressor)
    from sklearn.linear_model import LinearRegression
    from sklearn.model_selection import train_test_split
    from mlprodict.onnxrt import OnnxInference
    from onnxruntime import InferenceSession
    from skl2onnx import to_onnx
    from onnxcustom.utils import measure_time


    N = 11000
    X, y = make_regression(N, n_features=10)
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, train_size=0.01)
    print("Train shape", X_train.shape)
    print("Test shape", X_test.shape)

    reg1 = GradientBoostingRegressor(random_state=1)
    reg2 = RandomForestRegressor(random_state=1)
    reg3 = LinearRegression()
    ereg = VotingRegressor([('gb', reg1), ('rf', reg2), ('lr', reg3)])
    ereg.fit(X_train, y_train)





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    Train shape (110, 10)
    Test shape (10890, 10)

    VotingRegressor(estimators=[('gb', GradientBoostingRegressor(random_state=1)),
                                ('rf', RandomForestRegressor(random_state=1)),
                                ('lr', LinearRegression())])



.. GENERATED FROM PYTHON SOURCE LINES 49-58

Measure the processing time
+++++++++++++++++++++++++++

We use function :func:`onnxcustom.utils.measure_time`.
The page about `assume_finite <https://scikit-learn.org/
stable/modules/generated/sklearn.config_context.html>`_
may be useful if you need to optimize the prediction.
We measure the processing time per observation whether
or not an observation belongs to a batch or is a single one.

.. GENERATED FROM PYTHON SOURCE LINES 58-75

.. code-block:: default


    sizes = [(1, 50), (10, 50), (1000, 10), (10000, 5)]

    with config_context(assume_finite=True):
        obs = []
        for batch_size, repeat in tqdm(sizes):
            context = {"ereg": ereg, 'X': X_test[:batch_size]}
            mt = measure_time(
                "ereg.predict(X)", context, div_by_number=True,
                number=10, repeat=repeat)
            mt['size'] = context['X'].shape[0]
            mt['mean_obs'] = mt['average'] / mt['size']
            obs.append(mt)

    df_skl = DataFrame(obs)
    df_skl





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none


      0%|          | 0/4 [00:00<?, ?it/s]
     25%|##5       | 1/4 [00:23<01:09, 23.00s/it]
     50%|#####     | 2/4 [00:45<00:45, 22.87s/it]
     75%|#######5  | 3/4 [00:52<00:15, 15.39s/it]
    100%|##########| 4/4 [01:03<00:00, 13.86s/it]
    100%|##########| 4/4 [01:03<00:00, 15.95s/it]


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>average</th>
          <th>deviation</th>
          <th>min_exec</th>
          <th>max_exec</th>
          <th>repeat</th>
          <th>number</th>
          <th>size</th>
          <th>mean_obs</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.045983</td>
          <td>0.000208</td>
          <td>0.045755</td>
          <td>0.046876</td>
          <td>50</td>
          <td>10</td>
          <td>1</td>
          <td>0.045983</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.045538</td>
          <td>0.000105</td>
          <td>0.045349</td>
          <td>0.045724</td>
          <td>50</td>
          <td>10</td>
          <td>10</td>
          <td>0.004554</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.064788</td>
          <td>0.000276</td>
          <td>0.064599</td>
          <td>0.065574</td>
          <td>10</td>
          <td>10</td>
          <td>1000</td>
          <td>0.000065</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0.230436</td>
          <td>0.000118</td>
          <td>0.230340</td>
          <td>0.230657</td>
          <td>5</td>
          <td>10</td>
          <td>10000</td>
          <td>0.000023</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 76-77

Graphe.

.. GENERATED FROM PYTHON SOURCE LINES 77-81

.. code-block:: default


    df_skl.set_index('size')[['mean_obs']].plot(
        title="scikit-learn", logx=True, logy=True)




.. image:: /auto_examples/images/sphx_glr_plot_bbegin_measure_time_001.png
    :alt: scikit-learn
    :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 82-87

ONNX runtime
++++++++++++

The same is done with the two ONNX runtime
available.

.. GENERATED FROM PYTHON SOURCE LINES 87-124

.. code-block:: default


    onx = to_onnx(ereg, X_train[:1].astype(numpy.float32))
    sess = InferenceSession(onx.SerializeToString())
    oinf = OnnxInference(onx, runtime="python_compiled")

    obs = []
    for batch_size, repeat in tqdm(sizes):

        # scikit-learn
        context = {"ereg": ereg, 'X': X_test[:batch_size].astype(numpy.float32)}
        mt = measure_time(
            "ereg.predict(X)", context, div_by_number=True,
            number=10, repeat=repeat)
        mt['size'] = context['X'].shape[0]
        mt['skl'] = mt['average'] / mt['size']

        # onnxruntime
        context = {"sess": sess, 'X': X_test[:batch_size].astype(numpy.float32)}
        mt2 = measure_time(
            "sess.run(None, {'X': X})[0]", context, div_by_number=True,
            number=10, repeat=repeat)
        mt['ort'] = mt2['average'] / mt['size']

        # mlprodict
        context = {"oinf": oinf, 'X': X_test[:batch_size].astype(numpy.float32)}
        mt2 = measure_time(
            "oinf.run({'X': X})['variable']", context, div_by_number=True,
            number=10, repeat=repeat)
        mt['pyrt'] = mt2['average'] / mt['size']

        # end
        obs.append(mt)


    df = DataFrame(obs)
    df





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none


      0%|          | 0/4 [00:00<?, ?it/s]
     25%|##5       | 1/4 [00:24<01:14, 24.74s/it]
     50%|#####     | 2/4 [00:50<00:50, 25.24s/it]
     75%|#######5  | 3/4 [01:11<00:23, 23.51s/it]
    100%|##########| 4/4 [01:42<00:00, 26.28s/it]
    100%|##########| 4/4 [01:42<00:00, 25.58s/it]


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>average</th>
          <th>deviation</th>
          <th>min_exec</th>
          <th>max_exec</th>
          <th>repeat</th>
          <th>number</th>
          <th>size</th>
          <th>skl</th>
          <th>ort</th>
          <th>pyrt</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.046501</td>
          <td>0.000190</td>
          <td>0.046270</td>
          <td>0.047331</td>
          <td>50</td>
          <td>10</td>
          <td>1</td>
          <td>0.046501</td>
          <td>0.000332</td>
          <td>0.002612</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.046235</td>
          <td>0.000145</td>
          <td>0.046030</td>
          <td>0.046850</td>
          <td>50</td>
          <td>10</td>
          <td>10</td>
          <td>0.004624</td>
          <td>0.000143</td>
          <td>0.000348</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.065743</td>
          <td>0.000228</td>
          <td>0.065444</td>
          <td>0.066315</td>
          <td>10</td>
          <td>10</td>
          <td>1000</td>
          <td>0.000066</td>
          <td>0.000009</td>
          <td>0.000140</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0.233349</td>
          <td>0.001563</td>
          <td>0.231904</td>
          <td>0.236348</td>
          <td>5</td>
          <td>10</td>
          <td>10000</td>
          <td>0.000023</td>
          <td>0.000006</td>
          <td>0.000031</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 125-126

Graph.

.. GENERATED FROM PYTHON SOURCE LINES 126-131

.. code-block:: default


    df.set_index('size')[['skl', 'ort', 'pyrt']].plot(
        title="Average prediction time per runtime",
        logx=True, logy=True)




.. image:: /auto_examples/images/sphx_glr_plot_bbegin_measure_time_002.png
    :alt: Average prediction time per runtime
    :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 132-138

:epkg:`ONNX` runtimes are much faster than :epkg:`scikit-learn`
to predict one observation. :epkg:`scikit-learn` is optimized
for training, for batch prediction. That explains why
:epkg:`scikit-learn` and ONNX runtimes seem to converge
for big batches. They use similar implementation,
parallelization and languages (:epkg:`C++`, :epkg:`openmp`).


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 3 minutes  3.698 seconds)


.. _sphx_glr_download_auto_examples_plot_bbegin_measure_time.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example


  .. container:: binder-badge

    .. image:: images/binder_badge_logo.svg
      :target: https://mybinder.org/v2/gh/sdpython/onnxcustom/master?urlpath=lab/tree/notebooks/auto_examples/plot_bbegin_measure_time.ipynb
      :alt: Launch binder
      :width: 150 px


  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: plot_bbegin_measure_time.py <plot_bbegin_measure_time.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: plot_bbegin_measure_time.ipynb <plot_bbegin_measure_time.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_