.. _neuraltreecostrst:

=====================
NeuralTreeNet et coût
=====================


.. only:: html

    **Links:** :download:`notebook <neural_tree_cost.ipynb>`, :downloadlink:`html <neural_tree_cost2html.html>`, :download:`PDF <neural_tree_cost.pdf>`, :download:`python <neural_tree_cost.py>`, :downloadlink:`slides <neural_tree_cost.slides.html>`, :githublink:`GitHub|_doc/notebooks/ml/neural_tree_cost.ipynb|*`


La classe `NeuralTreeNet <:ref:%60cl-NeuralTreeNet%60>`__ convertit un
arbre de décision en réseau de neurones. Si la conversion n’est pas
exacte mais elle permet d’obtenir un modèle différentiable et apprenable
avec un algorithme d’optimisation à base de gradient. Ce notebook
compare le temps d’éxécution entre un arbre et le réseau de neurones.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





.. code:: ipython3

    %matplotlib inline

.. code:: ipython3

    %load_ext mlprodict

Jeux de données
---------------

On construit un jeu de données aléatoire.

.. code:: ipython3

    import numpy
    
    X = numpy.random.randn(10000, 10)
    y = X.sum(axis=1) / X.shape[1]
    X = X.astype(numpy.float64)
    y = y.astype(numpy.float64)

.. code:: ipython3

    middle = X.shape[0] // 2
    X_train, X_test = X[:middle], X[middle:]
    y_train, y_test = y[:middle], y[middle:]

Caler un arbre de décision
--------------------------

.. code:: ipython3

    from sklearn.tree import DecisionTreeRegressor
    
    tree = DecisionTreeRegressor(max_depth=7)
    tree.fit(X_train, y_train)
    tree.score(X_train, y_train), tree.score(X_test, y_test)




.. parsed-literal::
    (0.6091161526814477, 0.3884519134946681)



.. code:: ipython3

    from sklearn.metrics import r2_score
    r2_score(y_test, tree.predict(X_test))




.. parsed-literal::
    0.3884519134946681



Covnersion de l’arbre en réseau de neurones

.. code:: ipython3

    from pandas import DataFrame
    from mlstatpy.ml.neural_tree import NeuralTreeNet, NeuralTreeNetRegressor
    
    xe = X_test.astype(numpy.float32)
    expected = tree.predict(xe)
    
    nn = NeuralTreeNetRegressor(NeuralTreeNet.create_from_tree(tree, arch='compact'))
    got = nn.predict(xe)
    me = numpy.abs(got - expected).mean()
    mx = numpy.abs(got - expected).max()
    DataFrame([{"average absolute error": me, "max absolute error": mx}]).T






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>average absolute error</th>
          <td>0.210314</td>
        </tr>
        <tr>
          <th>max absolute error</th>
          <td>1.679553</td>
        </tr>
      </tbody>
    </table>
    </div>



La conversion est loin d’être parfaite. La raison vient du fait que les
fonctions de seuil sont approchées par des fonctions sigmoïdes. Il
suffit d’une erreur minime pour que la décision prenne un chemin
différent dans l’arbre et soit complètement différente.

Conversion au format ONNX
-------------------------

.. code:: ipython3

    from mlprodict.onnx_conv import to_onnx
    
    onx_tree = to_onnx(tree, X[:1].astype(numpy.float32))
    onx_nn = to_onnx(nn, X[:1].astype(numpy.float32))

Le réseau de neurones peut être représenté comme suit.

.. code:: ipython3

    %onnxview onx_nn






.. raw:: html

    <div id="M1f0c5529ea564ba2bdf10042b00fd6b1-cont"><div id="M1f0c5529ea564ba2bdf10042b00fd6b1" style="width:;height:;"></div></div>
    <script>

    require(['http://www.xavierdupre.fr/js/vizjs/viz.js'], function() { var svgGraph = Viz("digraph{\n  nodesep=0.05;\n  ranksep=0.25;\n  size=7;\n  orientation=portrait;\n\n  X [shape=box color=red label=\"X\nfloat((0, 10))\" fontsize=10];\n\n  variable [shape=box color=green label=\"variable\nfloat((0, 1))\" fontsize=10];\n\n  Ma_MatMulcst [shape=box label=\"Ma_MatMulcst\nfloat32((10, 127))\n[[ 0.  0.  0. ...  0.  0.  0.]\n [ 0.  0.  0. ...  ...\" fontsize=10];\n  Ad_Addcst [shape=box label=\"Ad_Addcst\nfloat32((127,))\n[-0.28751326 -0.3320096   0.24225289 -1.2377734   ...\" fontsize=10];\n  Mu_Mulcst [shape=box label=\"Mu_Mulcst\nfloat32((1,))\n[4.]\" fontsize=10];\n  Ma_MatMulcst1 [shape=box label=\"Ma_MatMulcst1\nfloat32((127, 128))\n[[ 1.  1.  1. ... -1. -1. -1.]\n [ 1.  1.  1. ...  ...\" fontsize=10];\n  Ad_Addcst1 [shape=box label=\"Ad_Addcst1\nfloat32((128,))\n[-6.5 -5.5 -5.5 -4.5 -5.5 -4.5 -4.5 -3.5 -5.5 -4.5...\" fontsize=10];\n  Ma_MatMulcst2 [shape=box label=\"Ma_MatMulcst2\nfloat32((128, 1))\n[[-1.0975432 ]\n [-0.9416567 ]\n [-0.69355935]\n [-0....\" fontsize=10];\n  Ad_Addcst2 [shape=box label=\"Ad_Addcst2\nfloat32((1,))\n[0.]\" fontsize=10];\n\n  Ma_Y02 [shape=box label=\"Ma_Y02\" fontsize=10];\n  Ma_MatMul [shape=box style=\"filled,rounded\" color=orange label=\"MatMul\n(Ma_MatMul)\" fontsize=10];\n  X -> Ma_MatMul;\n  Ma_MatMulcst -> Ma_MatMul;\n  Ma_MatMul -> Ma_Y02;\n\n  Ad_C02 [shape=box label=\"Ad_C02\" fontsize=10];\n  Ad_Add [shape=box style=\"filled,rounded\" color=orange label=\"Add\n(Ad_Add)\" fontsize=10];\n  Ma_Y02 -> Ad_Add;\n  Ad_Addcst -> Ad_Add;\n  Ad_Add -> Ad_C02;\n\n  Mu_C01 [shape=box label=\"Mu_C01\" fontsize=10];\n  Mu_Mul [shape=box style=\"filled,rounded\" color=orange label=\"Mul\n(Mu_Mul)\" fontsize=10];\n  Ad_C02 -> Mu_Mul;\n  Mu_Mulcst -> Mu_Mul;\n  Mu_Mul -> Mu_C01;\n\n  Si_Y01 [shape=box label=\"Si_Y01\" fontsize=10];\n  Si_Sigmoid [shape=box style=\"filled,rounded\" color=orange label=\"Sigmoid\n(Si_Sigmoid)\" fontsize=10];\n  Mu_C01 -> Si_Sigmoid;\n  Si_Sigmoid -> Si_Y01;\n\n  Ma_Y01 [shape=box label=\"Ma_Y01\" fontsize=10];\n  Ma_MatMul1 [shape=box style=\"filled,rounded\" color=orange label=\"MatMul\n(Ma_MatMul1)\" fontsize=10];\n  Si_Y01 -> Ma_MatMul1;\n  Ma_MatMulcst1 -> Ma_MatMul1;\n  Ma_MatMul1 -> Ma_Y01;\n\n  Ad_C01 [shape=box label=\"Ad_C01\" fontsize=10];\n  Ad_Add1 [shape=box style=\"filled,rounded\" color=orange label=\"Add\n(Ad_Add1)\" fontsize=10];\n  Ma_Y01 -> Ad_Add1;\n  Ad_Addcst1 -> Ad_Add1;\n  Ad_Add1 -> Ad_C01;\n\n  Mu_C0 [shape=box label=\"Mu_C0\" fontsize=10];\n  Mu_Mul1 [shape=box style=\"filled,rounded\" color=orange label=\"Mul\n(Mu_Mul1)\" fontsize=10];\n  Ad_C01 -> Mu_Mul1;\n  Mu_Mulcst -> Mu_Mul1;\n  Mu_Mul1 -> Mu_C0;\n\n  Si_Y0 [shape=box label=\"Si_Y0\" fontsize=10];\n  Si_Sigmoid1 [shape=box style=\"filled,rounded\" color=orange label=\"Sigmoid\n(Si_Sigmoid1)\" fontsize=10];\n  Mu_C0 -> Si_Sigmoid1;\n  Si_Sigmoid1 -> Si_Y0;\n\n  Ma_Y0 [shape=box label=\"Ma_Y0\" fontsize=10];\n  Ma_MatMul2 [shape=box style=\"filled,rounded\" color=orange label=\"MatMul\n(Ma_MatMul2)\" fontsize=10];\n  Si_Y0 -> Ma_MatMul2;\n  Ma_MatMulcst2 -> Ma_MatMul2;\n  Ma_MatMul2 -> Ma_Y0;\n\n  Ad_C0 [shape=box label=\"Ad_C0\" fontsize=10];\n  Ad_Add2 [shape=box style=\"filled,rounded\" color=orange label=\"Add\n(Ad_Add2)\" fontsize=10];\n  Ma_Y0 -> Ad_Add2;\n  Ad_Addcst2 -> Ad_Add2;\n  Ad_Add2 -> Ad_C0;\n\n  Id_Identity1 [shape=box style=\"filled,rounded\" color=orange label=\"Identity\n(Id_Identity1)\" fontsize=10];\n  Ad_C0 -> Id_Identity1;\n  Id_Identity1 -> variable;\n}");
    document.getElementById('M1f0c5529ea564ba2bdf10042b00fd6b1').innerHTML = svgGraph; });

    </script>



.. code:: ipython3

    from mlprodict.plotting.text_plot import onnx_simple_text_plot
    print(onnx_simple_text_plot(onx_nn))


.. parsed-literal::
    opset: domain='' version=15
    input: name='X' type=dtype('float32') shape=[None, 10]
    init: name='Ma_MatMulcst' type=dtype('float32') shape=(1270,)
    init: name='Ad_Addcst' type=dtype('float32') shape=(127,)
    init: name='Mu_Mulcst' type=dtype('float32') shape=(1,) -- array([4.], dtype=float32)
    init: name='Ma_MatMulcst1' type=dtype('float32') shape=(16256,)
    init: name='Ad_Addcst1' type=dtype('float32') shape=(128,)
    init: name='Ma_MatMulcst2' type=dtype('float32') shape=(128,)
    init: name='Ad_Addcst2' type=dtype('float32') shape=(1,) -- array([0.], dtype=float32)
    MatMul(X, Ma_MatMulcst) -> Ma_Y02
      Add(Ma_Y02, Ad_Addcst) -> Ad_C02
        Mul(Ad_C02, Mu_Mulcst) -> Mu_C01
          Sigmoid(Mu_C01) -> Si_Y01
            MatMul(Si_Y01, Ma_MatMulcst1) -> Ma_Y01
              Add(Ma_Y01, Ad_Addcst1) -> Ad_C01
                Mul(Ad_C01, Mu_Mulcst) -> Mu_C0
                  Sigmoid(Mu_C0) -> Si_Y0
                    MatMul(Si_Y0, Ma_MatMulcst2) -> Ma_Y0
                      Add(Ma_Y0, Ad_Addcst2) -> Ad_C0
                        Identity(Ad_C0) -> variable
    output: name='variable' type=dtype('float32') shape=[None, 1]


Temps de calcul des prédictions
-------------------------------

.. code:: ipython3

    from mlprodict.onnxrt import OnnxInference
    
    oinf_tree = OnnxInference(onx_tree, runtime='onnxruntime1')
    oinf_nn = OnnxInference(onx_nn, runtime='onnxruntime1')
    
    %timeit tree.predict(xe)


.. parsed-literal::
    No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5'
    518 µs ± 41.4 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)


.. code:: ipython3

    %timeit oinf_tree.run({'X': xe})


.. parsed-literal::
    124 µs ± 5.21 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)


.. code:: ipython3

    %timeit oinf_nn.run({'X': xe})


.. parsed-literal::
    3.18 ms ± 569 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)


Le temps de calcul est nettement plus long pour le réseau de neurones.
Si l’arbre de décision a une profondeur de *d*, l’arbre de décision va
faire exactement *d* comparaisons. Le réseau de neurones quant à lui
évalue tous les seuils pour chaque prédiction, soit :math:`2^d`.
Vérifions cela en faisant variable la profondeur.

Temps de calcul en fonction de la profondeur
--------------------------------------------

.. code:: ipython3

    from tqdm import tqdm
    from cpyquickhelper.numbers import measure_time
    
    data = []
    for d in tqdm(range(2, 10)):
        tree = DecisionTreeRegressor(max_depth=d)
        tree.fit(X_train, y_train)
        obs = measure_time(lambda: tree.predict(xe), number=20, repeat=20)
        obs.update(dict(d=d, exp='skl'))
        data.append(obs)
        
        nn = NeuralTreeNetRegressor(NeuralTreeNet.create_from_tree(tree, arch='compact'))
        
        onx_tree = to_onnx(tree, X[:1].astype(numpy.float32))
        onx_nn = to_onnx(nn, X[:1].astype(numpy.float32))
        oinf_tree = OnnxInference(onx_tree, runtime='onnxruntime1')
        oinf_nn = OnnxInference(onx_nn, runtime='onnxruntime1')
        
        obs = measure_time(lambda: oinf_tree.run({'X': xe}), number=10, repeat=10)
        obs.update(dict(d=d, exp='onx_tree'))
        data.append(obs)
    
        obs = measure_time(lambda: oinf_nn.run({'X': xe}), number=10, repeat=10)
        obs.update(dict(d=d, exp='onx_nn'))
        data.append(obs)
    
    df = DataFrame(data)
    df


.. parsed-literal::
    100%|██████████| 8/8 [00:07<00:00,  1.04it/s]






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>average</th>
          <th>deviation</th>
          <th>min_exec</th>
          <th>max_exec</th>
          <th>repeat</th>
          <th>number</th>
          <th>ttime</th>
          <th>context_size</th>
          <th>d</th>
          <th>exp</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.005656</td>
          <td>0.001292</td>
          <td>0.004990</td>
          <td>0.009533</td>
          <td>20</td>
          <td>20</td>
          <td>0.113116</td>
          <td>64</td>
          <td>2</td>
          <td>skl</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.000609</td>
          <td>0.000084</td>
          <td>0.000538</td>
          <td>0.000830</td>
          <td>10</td>
          <td>10</td>
          <td>0.006088</td>
          <td>64</td>
          <td>2</td>
          <td>onx_tree</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.002337</td>
          <td>0.000563</td>
          <td>0.001737</td>
          <td>0.003299</td>
          <td>10</td>
          <td>10</td>
          <td>0.023374</td>
          <td>64</td>
          <td>2</td>
          <td>onx_nn</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0.005607</td>
          <td>0.000962</td>
          <td>0.004669</td>
          <td>0.007374</td>
          <td>20</td>
          <td>20</td>
          <td>0.112131</td>
          <td>64</td>
          <td>3</td>
          <td>skl</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0.000709</td>
          <td>0.000240</td>
          <td>0.000534</td>
          <td>0.001363</td>
          <td>10</td>
          <td>10</td>
          <td>0.007087</td>
          <td>64</td>
          <td>3</td>
          <td>onx_tree</td>
        </tr>
        <tr>
          <th>5</th>
          <td>0.002506</td>
          <td>0.000448</td>
          <td>0.002073</td>
          <td>0.003736</td>
          <td>10</td>
          <td>10</td>
          <td>0.025064</td>
          <td>64</td>
          <td>3</td>
          <td>onx_nn</td>
        </tr>
        <tr>
          <th>6</th>
          <td>0.006398</td>
          <td>0.000942</td>
          <td>0.005576</td>
          <td>0.009556</td>
          <td>20</td>
          <td>20</td>
          <td>0.127961</td>
          <td>64</td>
          <td>4</td>
          <td>skl</td>
        </tr>
        <tr>
          <th>7</th>
          <td>0.000852</td>
          <td>0.000335</td>
          <td>0.000633</td>
          <td>0.001715</td>
          <td>10</td>
          <td>10</td>
          <td>0.008519</td>
          <td>64</td>
          <td>4</td>
          <td>onx_tree</td>
        </tr>
        <tr>
          <th>8</th>
          <td>0.004151</td>
          <td>0.000763</td>
          <td>0.003408</td>
          <td>0.005826</td>
          <td>10</td>
          <td>10</td>
          <td>0.041507</td>
          <td>64</td>
          <td>4</td>
          <td>onx_nn</td>
        </tr>
        <tr>
          <th>9</th>
          <td>0.007266</td>
          <td>0.000731</td>
          <td>0.006400</td>
          <td>0.009247</td>
          <td>20</td>
          <td>20</td>
          <td>0.145314</td>
          <td>64</td>
          <td>5</td>
          <td>skl</td>
        </tr>
        <tr>
          <th>10</th>
          <td>0.000963</td>
          <td>0.000169</td>
          <td>0.000826</td>
          <td>0.001327</td>
          <td>10</td>
          <td>10</td>
          <td>0.009634</td>
          <td>64</td>
          <td>5</td>
          <td>onx_tree</td>
        </tr>
        <tr>
          <th>11</th>
          <td>0.006113</td>
          <td>0.000560</td>
          <td>0.005360</td>
          <td>0.007255</td>
          <td>10</td>
          <td>10</td>
          <td>0.061134</td>
          <td>64</td>
          <td>5</td>
          <td>onx_nn</td>
        </tr>
        <tr>
          <th>12</th>
          <td>0.008881</td>
          <td>0.001234</td>
          <td>0.007601</td>
          <td>0.012340</td>
          <td>20</td>
          <td>20</td>
          <td>0.177622</td>
          <td>64</td>
          <td>6</td>
          <td>skl</td>
        </tr>
        <tr>
          <th>13</th>
          <td>0.001175</td>
          <td>0.000613</td>
          <td>0.000815</td>
          <td>0.002984</td>
          <td>10</td>
          <td>10</td>
          <td>0.011754</td>
          <td>64</td>
          <td>6</td>
          <td>onx_tree</td>
        </tr>
        <tr>
          <th>14</th>
          <td>0.011537</td>
          <td>0.001079</td>
          <td>0.010146</td>
          <td>0.013582</td>
          <td>10</td>
          <td>10</td>
          <td>0.115367</td>
          <td>64</td>
          <td>6</td>
          <td>onx_nn</td>
        </tr>
        <tr>
          <th>15</th>
          <td>0.011134</td>
          <td>0.000887</td>
          <td>0.009569</td>
          <td>0.013273</td>
          <td>20</td>
          <td>20</td>
          <td>0.222676</td>
          <td>64</td>
          <td>7</td>
          <td>skl</td>
        </tr>
        <tr>
          <th>16</th>
          <td>0.001399</td>
          <td>0.000131</td>
          <td>0.001301</td>
          <td>0.001742</td>
          <td>10</td>
          <td>10</td>
          <td>0.013988</td>
          <td>64</td>
          <td>7</td>
          <td>onx_tree</td>
        </tr>
        <tr>
          <th>17</th>
          <td>0.034831</td>
          <td>0.004986</td>
          <td>0.027853</td>
          <td>0.046760</td>
          <td>10</td>
          <td>10</td>
          <td>0.348306</td>
          <td>64</td>
          <td>7</td>
          <td>onx_nn</td>
        </tr>
        <tr>
          <th>18</th>
          <td>0.011263</td>
          <td>0.001060</td>
          <td>0.009911</td>
          <td>0.013893</td>
          <td>20</td>
          <td>20</td>
          <td>0.225251</td>
          <td>64</td>
          <td>8</td>
          <td>skl</td>
        </tr>
        <tr>
          <th>19</th>
          <td>0.001226</td>
          <td>0.000148</td>
          <td>0.001033</td>
          <td>0.001609</td>
          <td>10</td>
          <td>10</td>
          <td>0.012259</td>
          <td>64</td>
          <td>8</td>
          <td>onx_tree</td>
        </tr>
        <tr>
          <th>20</th>
          <td>0.141722</td>
          <td>0.051041</td>
          <td>0.089979</td>
          <td>0.270129</td>
          <td>10</td>
          <td>10</td>
          <td>1.417218</td>
          <td>64</td>
          <td>8</td>
          <td>onx_nn</td>
        </tr>
        <tr>
          <th>21</th>
          <td>0.016436</td>
          <td>0.004827</td>
          <td>0.011379</td>
          <td>0.028261</td>
          <td>20</td>
          <td>20</td>
          <td>0.328716</td>
          <td>64</td>
          <td>9</td>
          <td>skl</td>
        </tr>
        <tr>
          <th>22</th>
          <td>0.003879</td>
          <td>0.001963</td>
          <td>0.002402</td>
          <td>0.008579</td>
          <td>10</td>
          <td>10</td>
          <td>0.038786</td>
          <td>64</td>
          <td>9</td>
          <td>onx_tree</td>
        </tr>
        <tr>
          <th>23</th>
          <td>0.323764</td>
          <td>0.063045</td>
          <td>0.261038</td>
          <td>0.456420</td>
          <td>10</td>
          <td>10</td>
          <td>3.237635</td>
          <td>64</td>
          <td>9</td>
          <td>onx_nn</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    piv = df.pivot('d', 'exp', 'average')
    piv.plot(logy=True, title="Temps de calcul en fonction de la profondeur");



.. image:: neural_tree_cost_25_0.png


L’hypothèse est vérifiée.