.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "gyexamples/plot_onnx_benchmark.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_gyexamples_plot_onnx_benchmark.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_gyexamples_plot_onnx_benchmark.py:


.. _l-example-onnx-benchmark:

Measure ONNX runtime performances
=================================

The following example shows how to use the
command line to compare one or two runtimes
with :epkg:`scikit-learn`.
It relies on function :func:`validate_runtime
<mlprodict.cli.validate_runtime>` which can be called
from *python* or through a command line
described in page :ref:`l-CMD2`.

.. contents::
    :local:

Run the benchmark
+++++++++++++++++

The following line creates a folder used to dump
information about models which failed during the benchmark.

.. GENERATED FROM PYTHON SOURCE LINES 24-31

.. code-block:: default

    import os
    import matplotlib.image as mpimg
    import matplotlib.pyplot as plt
    import pandas
    if not os.path.exists("dump_errors"):
        os.mkdir("dump_errors")








.. GENERATED FROM PYTHON SOURCE LINES 32-45

The benchmark can be run with a python instruction
or a command line:

::

  python -m mlprodict validate_runtime -v 1 --out_raw data.csv --out_summary summary.csv
             -b 1 --dump_folder dump_errors --runtime python,onnxruntime1
             --models LinearRegression,DecisionTreeRegressor
             --n_features 4,10 --out_graph bench_png
             -t "{\"1\":{\"number\":10,\"repeat\":10},\"10\":{\"number\":5,\"repeat\":5}}"

We use the python instruction in this example.


.. GENERATED FROM PYTHON SOURCE LINES 45-64

.. code-block:: default

    from mlprodict.cli import validate_runtime

    validate_runtime(
        verbose=1,
        out_raw="data.csv", out_summary="summary.csv",
        benchmark=True, dump_folder="dump_errors",
        runtime=['python', 'onnxruntime1'],
        models=['LinearRegression', 'DecisionTreeRegressor'],
        n_features=[4, 10], dtype="32",
        out_graph="bench.png",
        time_kwargs={
            1: {"number": 100, "repeat": 100},
            10: {"number": 50, "repeat": 50},
            100: {"number": 40, "repeat": 50},
            1000: {"number": 40, "repeat": 40},
            10000: {"number": 20, "repeat": 20},
        }
    )




.. image-sg:: /gyexamples/images/sphx_glr_plot_onnx_benchmark_001.png
   :alt: RT/SKL-N=1, N=10, N=100, N=1000, N=10000
   :srcset: /gyexamples/images/sphx_glr_plot_onnx_benchmark_001.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    time_kwargs={1: {'number': 100, 'repeat': 100}, 10: {'number': 50, 'repeat': 50}, 100: {'number': 40, 'repeat': 50}, 1000: {'number': 40, 'repeat': 40}, 10000: {'number': 20, 'repeat': 20}}
    [enumerate_validated_operator_opsets] opset in [17, None].

      0%|          | 0/2 [00:00<?, ?it/s]
    LinearRegression            :   0%|          | 0/2 [00:00<?, ?it/s][enumerate_compatible_opset] opset in [17, None].

    LinearRegression            :  50%|#####     | 1/2 [03:11<03:11, 191.39s/it]
    DecisionTreeRegressor       :  50%|#####     | 1/2 [03:11<03:11, 191.39s/it][enumerate_compatible_opset] opset in [17, None].

    DecisionTreeRegressor       : 100%|##########| 2/2 [10:35<00:00, 340.33s/it]
    DecisionTreeRegressor       : 100%|##########| 2/2 [10:35<00:00, 317.99s/it]
    Saving raw_data into 'data.csv'.
    Saving summary into 'summary.csv'.
    Saving graph into 'bench.png'.
    findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['STIXSizeOneSym'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['STIXSizeTwoSym'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['STIXSizeThreeSym'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['STIXSizeFourSym'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['STIXSizeFiveSym'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['cmsy10'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['cmr10'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['cmtt10'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['cmmi10'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['cmb10'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['cmss10'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['cmex10'] not found. Falling back to DejaVu Sans.
    findfont: Font family ['DejaVu Sans Display'] not found. Falling back to DejaVu Sans.




.. GENERATED FROM PYTHON SOURCE LINES 65-66

Let's show the results.

.. GENERATED FROM PYTHON SOURCE LINES 66-69

.. code-block:: default

    df = pandas.read_csv("summary.csv")
    df.head(n=2).T






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
          <th>1</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>name</th>
          <td>DecisionTreeRegressor</td>
          <td>DecisionTreeRegressor</td>
        </tr>
        <tr>
          <th>problem</th>
          <td>b-reg</td>
          <td>b-reg</td>
        </tr>
        <tr>
          <th>scenario</th>
          <td>default</td>
          <td>default</td>
        </tr>
        <tr>
          <th>optim</th>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>method_name</th>
          <td>predict</td>
          <td>predict</td>
        </tr>
        <tr>
          <th>output_index</th>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>conv_options</th>
          <td>{}</td>
          <td>{}</td>
        </tr>
        <tr>
          <th>inst</th>
          <td>{"random_state": 42}</td>
          <td>{"random_state": 42}</td>
        </tr>
        <tr>
          <th>n_features</th>
          <td>4</td>
          <td>4</td>
        </tr>
        <tr>
          <th>runtime</th>
          <td>onnxruntime1</td>
          <td>python</td>
        </tr>
        <tr>
          <th>skl_version</th>
          <td>1.2.1</td>
          <td>1.2.1</td>
        </tr>
        <tr>
          <th>skl_nop</th>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>skl_ncoef</th>
          <td>-1.0</td>
          <td>-1.0</td>
        </tr>
        <tr>
          <th>skl_nlin</th>
          <td>-1.0</td>
          <td>-1.0</td>
        </tr>
        <tr>
          <th>skl_nnodes</th>
          <td>223.0</td>
          <td>223.0</td>
        </tr>
        <tr>
          <th>skl_ntrees</th>
          <td>1.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>skl_max_depth</th>
          <td>13.0</td>
          <td>13.0</td>
        </tr>
        <tr>
          <th>onx_size</th>
          <td>8943</td>
          <td>8943</td>
        </tr>
        <tr>
          <th>onx_nnodes</th>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>onx_ninits</th>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>onx_producer_name</th>
          <td>skl2onnx</td>
          <td>skl2onnx</td>
        </tr>
        <tr>
          <th>onx_producer_version</th>
          <td>1.13.1</td>
          <td>1.13.1</td>
        </tr>
        <tr>
          <th>onx_ai.onnx.ml</th>
          <td>3</td>
          <td>3</td>
        </tr>
        <tr>
          <th>onx_size_optim</th>
          <td>8943</td>
          <td>8943</td>
        </tr>
        <tr>
          <th>onx_nnodes_optim</th>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>onx_ninits_optim</th>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>opset17</th>
          <td>OK 17/3</td>
          <td>OK 17/3</td>
        </tr>
        <tr>
          <th>RT/SKL-N=1</th>
          <td>0.724179</td>
          <td>0.311947</td>
        </tr>
        <tr>
          <th>N=10</th>
          <td>0.730205</td>
          <td>0.316873</td>
        </tr>
        <tr>
          <th>N=100</th>
          <td>0.762137</td>
          <td>31.396165</td>
        </tr>
        <tr>
          <th>N=1000</th>
          <td>0.586674</td>
          <td>70.842404</td>
        </tr>
        <tr>
          <th>N=10000</th>
          <td>0.317246</td>
          <td>122.252102</td>
        </tr>
        <tr>
          <th>RT/SKL-N=1-min</th>
          <td>0.673482</td>
          <td>0.303278</td>
        </tr>
        <tr>
          <th>RT/SKL-N=1-max</th>
          <td>1.126067</td>
          <td>0.316214</td>
        </tr>
        <tr>
          <th>N=10-min</th>
          <td>0.714238</td>
          <td>0.311731</td>
        </tr>
        <tr>
          <th>N=10-max</th>
          <td>0.740292</td>
          <td>0.322754</td>
        </tr>
        <tr>
          <th>N=100-min</th>
          <td>0.700777</td>
          <td>0.395965</td>
        </tr>
        <tr>
          <th>N=100-max</th>
          <td>2.601827</td>
          <td>75.713904</td>
        </tr>
        <tr>
          <th>N=1000-min</th>
          <td>0.56944</td>
          <td>0.728227</td>
        </tr>
        <tr>
          <th>N=1000-max</th>
          <td>0.667581</td>
          <td>311.160683</td>
        </tr>
        <tr>
          <th>N=10000-min</th>
          <td>0.312533</td>
          <td>40.025484</td>
        </tr>
        <tr>
          <th>N=10000-max</th>
          <td>0.32857</td>
          <td>587.093982</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 70-71

Let's display the graph generated by the function.

.. GENERATED FROM PYTHON SOURCE LINES 71-78

.. code-block:: default



    img = mpimg.imread('bench.png')
    fig = plt.imshow(img)
    fig.axes.get_xaxis().set_visible(False)
    fig.axes.get_yaxis().set_visible(False)
    plt.show()



.. image-sg:: /gyexamples/images/sphx_glr_plot_onnx_benchmark_002.png
   :alt: plot onnx benchmark
   :srcset: /gyexamples/images/sphx_glr_plot_onnx_benchmark_002.png
   :class: sphx-glr-single-img






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 10 minutes  43.597 seconds)


.. _sphx_glr_download_gyexamples_plot_onnx_benchmark.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_onnx_benchmark.py <plot_onnx_benchmark.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_onnx_benchmark.ipynb <plot_onnx_benchmark.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_