.. _lightgbmdoublerst:

===============================
Lightgbm, double, discrepencies
===============================


.. only:: html

    **Links:** :download:`notebook <lightgbm_double.ipynb>`, :downloadlink:`html <lightgbm_double2html.html>`, :download:`PDF <lightgbm_double.pdf>`, :download:`python <lightgbm_double.py>`, :downloadlink:`slides <lightgbm_double.slides.html>`, :githublink:`GitHub|_doc/notebooks/lightgbm_double.ipynb|*`


Discrepencies usually happens with
`lightgbm <https://lightgbm.readthedocs.io/en/latest/>`__ because its
code is used double to represent the threshold of trees as ONNX is using
float only. There is no way to fix this discrepencies unless the ONNX
implementation of trees is using double.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





.. code:: ipython3

    %load_ext mlprodict

Simple regression problem
-------------------------

Target *y* is multiplied by 10 to increase the absolute discrepencies.
Relative discrepencies should not change much.

.. code:: ipython3

    from sklearn.datasets import make_regression
    from sklearn.model_selection import train_test_split
    X, y = make_regression(2000, n_features=10)
    y *= 10
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)

.. code:: ipython3

    min(y), max(y)




.. parsed-literal::
    (-5645.317056441552, 5686.0775071009075)



Training a model
----------------

Let’s train many models to see how they behave.

.. code:: ipython3

    from sklearn.ensemble import RandomForestRegressor
    from sklearn.ensemble import GradientBoostingRegressor
    from sklearn.ensemble import HistGradientBoostingRegressor
    from lightgbm import LGBMRegressor
    from xgboost import XGBRegressor

.. code:: ipython3

    models = [
        RandomForestRegressor(n_estimators=50, max_depth=8),
        GradientBoostingRegressor(n_estimators=50, max_depth=8),
        HistGradientBoostingRegressor(max_iter=50, max_depth=8),
        LGBMRegressor(n_estimators=50, max_depth=8),
        XGBRegressor(n_estimators=50, max_depth=8),
    ]

.. code:: ipython3

    from tqdm import tqdm
    for model in tqdm(models):
        model.fit(X_train, y_train)


.. parsed-literal::
    100%|██████████| 5/5 [00:01<00:00,  3.96it/s]


Conversion to ONNX
------------------

We use function *to_onnx* from this package to avoid the trouble of
registering converters from *onnxmltools* for *lightgbm* and *xgboost*
libraries.

.. code:: ipython3

    from mlprodict.onnx_conv import to_onnx
    import numpy
    onnx_models = [to_onnx(model, X_train[:1].astype(numpy.float32), rewrite_ops=True)
                   for model in models]


.. parsed-literal::
    C:\xavierdupre\__home_\github_fork\scikit-learn\sklearn\utils\deprecation.py:101: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use 'n_features_in_' instead.
      warnings.warn(msg, category=FutureWarning)
    C:\xavierdupre\__home_\github_fork\scikit-learn\sklearn\utils\deprecation.py:101: FutureWarning: Attribute n_classes_ was deprecated in version 0.24 and will be removed in 1.1 (renaming of 0.26).
      warnings.warn(msg, category=FutureWarning)


.. code:: ipython3

    simple_onx = to_onnx(LGBMRegressor(n_estimators=3, max_depth=4).fit(X_train, y_train),
                         X_train[:1].astype(numpy.float32), rewrite_ops=True)
    %onnxview simple_onx






.. raw:: html

    <div id="M2fe559de0a2442248dd225087eb42196-cont"><div id="M2fe559de0a2442248dd225087eb42196" style="width:100%;height:100%;"></div></div>
    <script>

    require(['http://www.xavierdupre.fr/js/vizjs/viz.js'], function() { var svgGraph = Viz("digraph{\n  orientation=portrait;\n  size=None;\n  nodesep=0.05;\n  ranksep=0.25;\n\n  X [shape=box color=red label=\"X\nfloat((0, 10))\" fontsize=10];\n\n  variable [shape=box color=green label=\"variable\nfloat((0, 1))\" fontsize=10];\n\n\n  LightGbmLGBMRegressor [shape=box style=\"filled,rounded\" color=orange label=\"TreeEnsembleRegressor\n(LightGbmLGBMRegressor)\nn_targets=1\nnodes_falsenodeids=[ 2  4 16  6...\nnodes_featureids=[6 9 9 5 2 0 1...\nnodes_hitrates=[1. 1. 1. 1. 1. ...\nnodes_missing_value_tracks_true=[1 1 1 1 1...\nnodes_modes=[b'BRANCH_LEQ' b'BR...\nnodes_nodeids=[ 0  1  2  3  4  ...\nnodes_treeids=[0 0 0 0 0 0 0 0 ...\nnodes_truenodeids=[ 1  3 15  5 ...\nnodes_values=[-0.3632275  -0.02...\npost_transform=b'NONE'\ntarget_ids=[0 0 0 0 0 0 0 0 0 0...\ntarget_nodeids=[ 5  7  8 11 12 ...\ntarget_treeids=[0 0 0 0 0 0 0 0...\ntarget_weights=[-295.27792  -20...\" fontsize=10];\n  X -> LightGbmLGBMRegressor;\n  LightGbmLGBMRegressor -> variable;\n}");
    document.getElementById('M2fe559de0a2442248dd225087eb42196').innerHTML = svgGraph; });

    </script>



Discrepencies with float32
--------------------------

.. code:: ipython3

    from onnxruntime import InferenceSession
    from pandas import DataFrame
    
    
    def max_discrepency(X, skl_model, onx_model):
        expected = skl_model.predict(X).ravel()
        
        sess = InferenceSession(onx_model.SerializeToString())
        got = sess.run(None, {'X': X})[0].ravel()
        
        diff = numpy.abs(got - expected).max()
        return diff
    
    
    obs = []
    x32 = X_test.astype(numpy.float32)
    for model, onx in zip(models, onnx_models):
        diff = max_discrepency(x32, model, onx)
        obs.append(dict(name=model.__class__.__name__, max_diff=diff))
    
        
    DataFrame(obs)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>name</th>
          <th>max_diff</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>RandomForestRegressor</td>
          <td>0.000493</td>
        </tr>
        <tr>
          <th>1</th>
          <td>GradientBoostingRegressor</td>
          <td>0.000937</td>
        </tr>
        <tr>
          <th>2</th>
          <td>HistGradientBoostingRegressor</td>
          <td>0.000794</td>
        </tr>
        <tr>
          <th>3</th>
          <td>LGBMRegressor</td>
          <td>0.000924</td>
        </tr>
        <tr>
          <th>4</th>
          <td>XGBRegressor</td>
          <td>0.000977</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    DataFrame(obs).set_index("name").plot(kind="bar").set_title("onnxruntime + float32");



.. image:: lightgbm_double_15_0.png


Discrepencies with mlprodict
----------------------------

This is not available with the current standard ONNX specifications. It
required *mlprodict* to implement a runtime for tree ensemble supporting
doubles.

.. code:: ipython3

    from mlprodict.onnxrt import OnnxInference
    from pandas import DataFrame
    
    
    def max_discrepency_2(X, skl_model, onx_model):
        expected = skl_model.predict(X).ravel()
        
        sess = OnnxInference(onx_model)
        got = sess.run({'X': X})['variable'].ravel()
        
        diff = numpy.abs(got - expected).max()
        return diff
    
    
    obs = []
    x32 = X_test.astype(numpy.float32)
    for model, onx in zip(models, onnx_models):
        diff = max_discrepency_2(x32, model, onx)
        obs.append(dict(name=model.__class__.__name__, max_diff=diff))
    
        
    DataFrame(obs)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>name</th>
          <th>max_diff</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>RandomForestRegressor</td>
          <td>0.000798</td>
        </tr>
        <tr>
          <th>1</th>
          <td>GradientBoostingRegressor</td>
          <td>0.001440</td>
        </tr>
        <tr>
          <th>2</th>
          <td>HistGradientBoostingRegressor</td>
          <td>0.001082</td>
        </tr>
        <tr>
          <th>3</th>
          <td>LGBMRegressor</td>
          <td>0.001288</td>
        </tr>
        <tr>
          <th>4</th>
          <td>XGBRegressor</td>
          <td>0.000122</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    DataFrame(obs).set_index("name").plot(kind="bar").set_title("mlprodict + float32");



.. image:: lightgbm_double_18_0.png


Discrepencies with mlprodict and double
---------------------------------------

The conversion needs to happen again.

.. code:: ipython3

    simple_onx = to_onnx(LGBMRegressor(n_estimators=2, max_depth=2).fit(X_train, y_train),
                         X_train[:1].astype(numpy.float64), rewrite_ops=True)
    %onnxview simple_onx


.. parsed-literal::
    C:\xavierdupre\microsoft_github\sklearn-onnx\skl2onnx\common\_container.py:603: UserWarning: Unable to find operator 'TreeEnsembleRegressorDouble' in domain 'mlprodict' in ONNX, op_version is forced to 1.
      warnings.warn(






.. raw:: html

    <div id="Mce0da6d95aad4720ba80e53a6e3c907c-cont"><div id="Mce0da6d95aad4720ba80e53a6e3c907c" style="width:100%;height:100%;"></div></div>
    <script>

    require(['http://www.xavierdupre.fr/js/vizjs/viz.js'], function() { var svgGraph = Viz("digraph{\n  orientation=portrait;\n  size=None;\n  nodesep=0.05;\n  ranksep=0.25;\n\n  X [shape=box color=red label=\"X\ndouble((0, 10))\" fontsize=10];\n\n  variable [shape=box color=green label=\"variable\ndouble((0, 1))\" fontsize=10];\n\n\n  LightGbmLGBMRegressor [shape=box style=\"filled,rounded\" color=orange label=\"TreeEnsembleRegressorDouble\n(LightGbmLGBMRegressor)\nn_targets=1\nnodes_falsenodeids=[2 4 6 0 0 0...\nnodes_featureids=[6 9 9 0 0 0 0...\nnodes_hitrates=[1. 1. 1. 1. 1. ...\nnodes_missing_value_tracks_true=[1 1 1 0 0...\nnodes_modes=[b'BRANCH_LEQ' b'BR...\nnodes_nodeids=[0 1 2 3 4 5 6 0 ...\nnodes_treeids=[0 0 0 0 0 0 0 1 ...\nnodes_truenodeids=[1 3 5 0 0 0 ...\nnodes_values=[-0.3632275  -0.02...\npost_transform=b'NONE'\ntarget_ids=[0 0 0 0 0 0 0 0]\ntarget_nodeids=[3 4 5 6 3 4 5 6...\ntarget_treeids=[0 0 0 0 1 1 1 1...\ntarget_weights=[-164.93028    -...\" fontsize=10];\n  X -> LightGbmLGBMRegressor;\n  LightGbmLGBMRegressor -> variable;\n}");
    document.getElementById('Mce0da6d95aad4720ba80e53a6e3c907c').innerHTML = svgGraph; });

    </script>



.. code:: ipython3

    onnx_models_64 = []
    for model in tqdm(models):
        onx = to_onnx(model, X_train[:1].astype(numpy.float64), rewrite_ops=True)
        onnx_models_64.append(onx)


.. parsed-literal::
      0%|          | 0/5 [00:00<?, ?it/s]C:\xavierdupre\__home_\github_fork\scikit-learn\sklearn\utils\deprecation.py:101: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use 'n_features_in_' instead.
      warnings.warn(msg, category=FutureWarning)
     20%|██        | 1/5 [00:02<00:09,  2.40s/it]C:\xavierdupre\__home_\github_fork\scikit-learn\sklearn\utils\deprecation.py:101: FutureWarning: Attribute n_classes_ was deprecated in version 0.24 and will be removed in 1.1 (renaming of 0.26).
      warnings.warn(msg, category=FutureWarning)
    100%|██████████| 5/5 [00:04<00:00,  1.16it/s]


.. code:: ipython3

    obs64 = []
    x64 = X_test.astype(numpy.float64)
    for model, onx in zip(models, onnx_models_64):
        oinf = OnnxInference(onx)
        diff = max_discrepency_2(x64, model, onx)
        obs64.append(dict(name=model.__class__.__name__, max_diff=diff))
    
        
    DataFrame(obs64)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>name</th>
          <th>max_diff</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>RandomForestRegressor</td>
          <td>2.273737e-12</td>
        </tr>
        <tr>
          <th>1</th>
          <td>GradientBoostingRegressor</td>
          <td>9.094947e-13</td>
        </tr>
        <tr>
          <th>2</th>
          <td>HistGradientBoostingRegressor</td>
          <td>9.094947e-13</td>
        </tr>
        <tr>
          <th>3</th>
          <td>LGBMRegressor</td>
          <td>4.686752e-05</td>
        </tr>
        <tr>
          <th>4</th>
          <td>XGBRegressor</td>
          <td>1.562066e-03</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    DataFrame(obs64).set_index("name").plot(kind="bar").set_title("mlprodict + float64");



.. image:: lightgbm_double_23_0.png


.. code:: ipython3

    df = DataFrame(obs).set_index('name').merge(DataFrame(obs64).set_index('name'),
                                                  left_index=True, right_index=True)
    df.columns = ['float32', 'float64']
    df






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>float32</th>
          <th>float64</th>
        </tr>
        <tr>
          <th>name</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>RandomForestRegressor</th>
          <td>0.000798</td>
          <td>2.273737e-12</td>
        </tr>
        <tr>
          <th>GradientBoostingRegressor</th>
          <td>0.001440</td>
          <td>9.094947e-13</td>
        </tr>
        <tr>
          <th>HistGradientBoostingRegressor</th>
          <td>0.001082</td>
          <td>9.094947e-13</td>
        </tr>
        <tr>
          <th>LGBMRegressor</th>
          <td>0.001288</td>
          <td>4.686752e-05</td>
        </tr>
        <tr>
          <th>XGBRegressor</th>
          <td>0.000122</td>
          <td>1.562066e-03</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    import matplotlib.pyplot as plt
    fig, ax = plt.subplots(1, 2, figsize=(12, 4))
    df.plot(kind="bar", ax=ax[0]).set_title("mlprodict")
    df.plot(kind="bar", ax=ax[1], logy=True).set_title("mlprodict");



.. image:: lightgbm_double_25_0.png


The runtime using double produces lower discrepencies except for
*xgboost*. It is probably using float and all the others are using
double.

**Note:** function
`to_onnx <http://www.xavierdupre.fr/app/mlprodict/helpsphinx/mlprodict/onnx_conv/convert.html#mlprodict.onnx_conv.convert.to_onnx>`__
automatically registers converters for *lightgbm*, *xgboost* and a
dedicated runtime for a new ONNX node
`TreeEnsembleRegressorDouble <http://www.xavierdupre.fr/app/mlprodict/helpsphinx/mlprodict/onnxrt/ops_cpu/op_tree_ensemble_regressor.html#mlprodict.onnxrt.ops_cpu.op_tree_ensemble_regressor.TreeEnsembleRegressorDouble>`__.
It uses
`skl2onnx.to_onnx <https://onnx.ai/sklearn-onnx/api_summary.html#skl2onnx.to_onnx>`__
underneath.