.. _mlcccmachinelearninginterpretabiliterst:

******************************************************
2A.ml - Interprétabilité et corrélations des variables
******************************************************


.. only:: html

    **Links:** :download:`notebook <ml_ccc_machine_learning_interpretabilite.ipynb>`, :downloadlink:`html <ml_ccc_machine_learning_interpretabilite2html.html>`, :download:`PDF <ml_ccc_machine_learning_interpretabilite.pdf>`, :download:`python <ml_ccc_machine_learning_interpretabilite.py>`, :downloadlink:`slides <ml_ccc_machine_learning_interpretabilite.slides.html>`, :githublink:`GitHub|_doc/notebooks/td2a_ml/ml_ccc_machine_learning_interpretabilite.ipynb|*`


Plus un modèle de machine learning contient de coefficients, moins sa
décision peut être interprétée. Comment contourner cet obstacle et
comprendre ce que le modèle a appris ? Notion de `feature
importance <http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html>`__.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





.. code:: ipython3

    # Répare une incompatibilité entre scipy 1.0 et statsmodels 0.8.
    from pymyinstall.fix import fix_scipy10_for_statsmodels08
    fix_scipy10_for_statsmodels08()

Modèles linéaires
-----------------

Les modèles linéaires sont les modèles les plus simples à interpréter. A
performance équivalente, il faut toujours choisir le modèle le plus
simple. Le module `scikit-learn <http://scikit-learn.org/>`__ ne propose
pas les outils standards d’analyse des modèles linéaires (test de
nullité, valeur propre). Il faut choisir
`statsmodels <http://statsmodels.sourceforge.net/>`__ pour obtenir ces
informations.

.. code:: ipython3

    import numpy
    import statsmodels.api as smapi
    nsample = 100
    x = numpy.linspace(0, 10, 100)
    X = numpy.column_stack((x, x**2 - x))
    beta = numpy.array([1, 0.1, 10])
    e = numpy.random.normal(size=nsample)
    X = smapi.add_constant(X)
    y = X @ beta + e

.. code:: ipython3

    model = smapi.OLS(y, X)
    results = model.fit()
    results.summary()






.. raw:: html

    <table class="simpletable">
    <caption>OLS Regression Results</caption>
    <tr>
      <th>Dep. Variable:</th>            <td>y</td>        <th>  R-squared:         </th> <td>   1.000</td> 
    </tr>
    <tr>
      <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   1.000</td> 
    </tr>
    <tr>
      <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>4.002e+06</td>
    </tr>
    <tr>
      <th>Date:</th>             <td>Sat, 25 Aug 2018</td> <th>  Prob (F-statistic):</th> <td>3.54e-239</td>
    </tr>
    <tr>
      <th>Time:</th>                 <td>17:18:49</td>     <th>  Log-Likelihood:    </th> <td> -137.02</td> 
    </tr>
    <tr>
      <th>No. Observations:</th>      <td>   100</td>      <th>  AIC:               </th> <td>   280.0</td> 
    </tr>
    <tr>
      <th>Df Residuals:</th>          <td>    97</td>      <th>  BIC:               </th> <td>   287.9</td> 
    </tr>
    <tr>
      <th>Df Model:</th>              <td>     2</td>      <th>                     </th>     <td> </td>    
    </tr>
    <tr>
      <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>    
    </tr>
    </table>
    <table class="simpletable">
    <tr>
        <td></td>       <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  
    </tr>
    <tr>
      <th>const</th> <td>    0.8264</td> <td>    0.284</td> <td>    2.906</td> <td> 0.005</td> <td>    0.262</td> <td>    1.391</td>
    </tr>
    <tr>
      <th>x1</th>    <td>    0.1508</td> <td>    0.119</td> <td>    1.265</td> <td> 0.209</td> <td>   -0.086</td> <td>    0.387</td>
    </tr>
    <tr>
      <th>x2</th>    <td>    9.9982</td> <td>    0.013</td> <td>  786.054</td> <td> 0.000</td> <td>    9.973</td> <td>   10.023</td>
    </tr>
    </table>
    <table class="simpletable">
    <tr>
      <th>Omnibus:</th>       <td> 0.492</td> <th>  Durbin-Watson:     </th> <td>   1.968</td>
    </tr>
    <tr>
      <th>Prob(Omnibus):</th> <td> 0.782</td> <th>  Jarque-Bera (JB):  </th> <td>   0.630</td>
    </tr>
    <tr>
      <th>Skew:</th>          <td> 0.046</td> <th>  Prob(JB):          </th> <td>   0.730</td>
    </tr>
    <tr>
      <th>Kurtosis:</th>      <td> 2.622</td> <th>  Cond. No.          </th> <td>    125.</td>
    </tr>
    </table><br/><br/>Warnings:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.



Arbres (tree)
-------------

Lectures
~~~~~~~~


-  `treeinterpreter <https://github.com/andosa/treeinterpreter>`__
-  `Making Tree Ensembles
   Interpretable <https://arxiv.org/pdf/1606.05390v1.pdf>`__ : l’article
   propose de simplifier une random forest en approximant sa sortie par
   une somme pondérée d’arbre plus simples.
-  `Understanding variable importances in forests of randomized
   trees <http://papers.nips.cc/paper/4928-understanding-variable-importances-in-forests-of-randomized-trees.pdf>`__
   : cet article explique plus formellement le calcul des termes
   ``feature_importances_`` calculés par scikit-learn pour chaque arbre
   et forêts d’arbres (voir aussi `Random Forests, Leo Breiman and Adele
   Cutler <http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm>`__)

Module treeinterpreter
~~~~~~~~~~~~~~~~~~~~~~

.. code:: ipython3

    from sklearn.datasets import load_iris
    iris = load_iris()
    X = iris.data
    Y = iris.target

.. code:: ipython3

    from sklearn.tree import DecisionTreeClassifier
    clf2 = DecisionTreeClassifier(max_depth=3)
    clf2.fit(X, Y)
    Yp2 = clf2.predict(X)

.. code:: ipython3

    from sklearn.tree import export_graphviz
    export_graphviz(clf2, out_file="arbre.dot")

.. code:: ipython3

    import os
    cwd = os.getcwd()
    from pyquickhelper.helpgen import find_graphviz_dot
    dot = find_graphviz_dot()
    os.system ("\"{1}\" -Tpng {0}\\arbre.dot -o {0}\\arbre.png".format(cwd, dot))




.. parsed-literal::
    0



.. code:: ipython3

    from IPython.display import Image
    Image("arbre.png")




.. image:: ml_ccc_machine_learning_interpretabilite_13_0.png



.. code:: ipython3

    from treeinterpreter import treeinterpreter
    pred, bias, contrib = treeinterpreter.predict(clf2, X[106:107,:])


.. parsed-literal::
    c:\python370_x64\lib\site-packages\sklearn\ensemble\weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
      from numpy.core.umath_tests import inner1d


.. code:: ipython3

    X[106:107,:]




.. parsed-literal::
    array([[4.9, 2.5, 4.5, 1.7]])



.. code:: ipython3

    pred




.. parsed-literal::
    array([[0.        , 0.97916667, 0.02083333]])



.. code:: ipython3

    bias




.. parsed-literal::
    array([[0.33333333, 0.33333333, 0.33333333]])



.. code:: ipython3

    contrib




.. parsed-literal::
    array([[[ 0.        ,  0.        ,  0.        ],
            [ 0.        ,  0.        ,  0.        ],
            [-0.33333333,  0.23842593,  0.09490741],
            [ 0.        ,  0.40740741, -0.40740741]]])



``pred`` est identique à ce que retourne la méthode ``predict`` de
scikit-learn. ``bias`` est la proportion de chaque classe. ``contrib``
est la somme des contributions de chaque variable à chaque classe. On
note :math:`X=(x_1, ..., x_n)` une observation.

.. math:: P(X \in classe(i)) = \sum_i contrib(x_k,i)

Le
`code <https://github.com/andosa/treeinterpreter/blob/master/treeinterpreter/treeinterpreter.py>`__
est assez facile à lire et permet de comprendre ce que vaut la fonction
:math:`contrib`.

Exercice 1 : décrire la fonction contrib
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

La lecture de `Understanding variable importances in forests of
randomized
trees <http://papers.nips.cc/paper/4928-understanding-variable-importances-in-forests-of-randomized-trees.pdf>`__
devrait vous y aider.

.. code:: ipython3

    clf2.feature_importances_




.. parsed-literal::
    array([0.        , 0.        , 0.58561555, 0.41438445])



Exercice 2 : implémenter l’algorithme
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Décrit dans `Making Tree Ensembles
Interpretable <https://arxiv.org/pdf/1606.05390v1.pdf>`__


Interprétation et corrélation
-----------------------------

Modèles linéaires
~~~~~~~~~~~~~~~~~

Les modèles linéaires n’aiment pas les variables corrélées. Dans
l’exemple qui suit, les variables :math:`X_2, X_3` sont identiques. La
régression ne peut retrouver les coefficients du modèle initial (2 et
8).

.. code:: ipython3

    import numpy
    import statsmodels.api as smapi
    nsample = 100
    x = numpy.linspace(0, 10, 100)
    X = numpy.column_stack((x, (x-5)**2, (x-5)**2))  # ajout de la même variable
    beta = numpy.array([1, 0.1, 2, 8])
    e = numpy.random.normal(size=nsample)
    X = smapi.add_constant(X)
    y = X @ beta + e

.. code:: ipython3

    import pandas
    pandas.DataFrame(numpy.corrcoef(X.T))


.. parsed-literal::
    c:\python370_x64\lib\site-packages\numpy\lib\function_base.py:2400: RuntimeWarning: invalid value encountered in true_divide
      c /= stddev[:, None]
    c:\python370_x64\lib\site-packages\numpy\lib\function_base.py:2401: RuntimeWarning: invalid value encountered in true_divide
      c /= stddev[None, :]






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
          <th>3</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>1</th>
          <td>NaN</td>
          <td>1.000000e+00</td>
          <td>-4.166739e-17</td>
          <td>-4.166739e-17</td>
        </tr>
        <tr>
          <th>2</th>
          <td>NaN</td>
          <td>-4.166739e-17</td>
          <td>1.000000e+00</td>
          <td>1.000000e+00</td>
        </tr>
        <tr>
          <th>3</th>
          <td>NaN</td>
          <td>-4.166739e-17</td>
          <td>1.000000e+00</td>
          <td>1.000000e+00</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    model = smapi.OLS(y, X)
    results = model.fit()
    results.summary()






.. raw:: html

    <table class="simpletable">
    <caption>OLS Regression Results</caption>
    <tr>
      <th>Dep. Variable:</th>            <td>y</td>        <th>  R-squared:         </th> <td>   1.000</td> 
    </tr>
    <tr>
      <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   1.000</td> 
    </tr>
    <tr>
      <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>2.786e+05</td>
    </tr>
    <tr>
      <th>Date:</th>             <td>Sat, 25 Aug 2018</td> <th>  Prob (F-statistic):</th> <td>4.67e-183</td>
    </tr>
    <tr>
      <th>Time:</th>                 <td>17:18:51</td>     <th>  Log-Likelihood:    </th> <td> -142.22</td> 
    </tr>
    <tr>
      <th>No. Observations:</th>      <td>   100</td>      <th>  AIC:               </th> <td>   290.4</td> 
    </tr>
    <tr>
      <th>Df Residuals:</th>          <td>    97</td>      <th>  BIC:               </th> <td>   298.3</td> 
    </tr>
    <tr>
      <th>Df Model:</th>              <td>     2</td>      <th>                     </th>     <td> </td>    
    </tr>
    <tr>
      <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>    
    </tr>
    </table>
    <table class="simpletable">
    <tr>
        <td></td>       <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  
    </tr>
    <tr>
      <th>const</th> <td>    0.9061</td> <td>    0.232</td> <td>    3.904</td> <td> 0.000</td> <td>    0.446</td> <td>    1.367</td>
    </tr>
    <tr>
      <th>x1</th>    <td>    0.1148</td> <td>    0.035</td> <td>    3.285</td> <td> 0.001</td> <td>    0.045</td> <td>    0.184</td>
    </tr>
    <tr>
      <th>x2</th>    <td>    5.0009</td> <td>    0.007</td> <td>  746.515</td> <td> 0.000</td> <td>    4.988</td> <td>    5.014</td>
    </tr>
    <tr>
      <th>x3</th>    <td>    5.0009</td> <td>    0.007</td> <td>  746.515</td> <td> 0.000</td> <td>    4.988</td> <td>    5.014</td>
    </tr>
    </table>
    <table class="simpletable">
    <tr>
      <th>Omnibus:</th>       <td> 0.790</td> <th>  Durbin-Watson:     </th> <td>   2.259</td>
    </tr>
    <tr>
      <th>Prob(Omnibus):</th> <td> 0.674</td> <th>  Jarque-Bera (JB):  </th> <td>   0.756</td>
    </tr>
    <tr>
      <th>Skew:</th>          <td>-0.206</td> <th>  Prob(JB):          </th> <td>   0.685</td>
    </tr>
    <tr>
      <th>Kurtosis:</th>      <td> 2.890</td> <th>  Cond. No.          </th> <td>4.06e+16</td>
    </tr>
    </table><br/><br/>Warnings:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.<br/>[2] The smallest eigenvalue is 1.67e-29. This might indicate that there are<br/>strong multicollinearity problems or that the design matrix is singular.



Arbre / tree
~~~~~~~~~~~~

Les arbres de décision n’aiment pas plus les variables corrélées.

.. code:: ipython3

    from sklearn.datasets import load_iris
    iris = load_iris()
    X = iris.data[:,:2]
    Y = iris.target

.. code:: ipython3

    from sklearn.tree import DecisionTreeClassifier
    clf1 = DecisionTreeClassifier(max_depth=3)
    clf1.fit(X, Y)




.. parsed-literal::
    DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=3,
                max_features=None, max_leaf_nodes=None,
                min_impurity_decrease=0.0, min_impurity_split=None,
                min_samples_leaf=1, min_samples_split=2,
                min_weight_fraction_leaf=0.0, presort=False, random_state=None,
                splitter='best')



.. code:: ipython3

    clf1.feature_importances_




.. parsed-literal::
    array([0.76759205, 0.23240795])



On recopie la variables :math:`X_1`.

.. code:: ipython3

    import numpy
    X2 = numpy.hstack([X, numpy.ones((X.shape[0], 1))])
    X2[:,2] = X2[:,0]

.. code:: ipython3

    clf2 = DecisionTreeClassifier(max_depth=3)
    clf2.fit(X2, Y)




.. parsed-literal::
    DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=3,
                max_features=None, max_leaf_nodes=None,
                min_impurity_decrease=0.0, min_impurity_split=None,
                min_samples_leaf=1, min_samples_split=2,
                min_weight_fraction_leaf=0.0, presort=False, random_state=None,
                splitter='best')



.. code:: ipython3

    clf2.feature_importances_




.. parsed-literal::
    array([0.21510471, 0.23240795, 0.55248735])



On voit que l’importance de la variable 1 est diluée sur deux variables.

Exercice 3 : variables corrélées pour un arbre de décision
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Un arbre de décision est composé d’un ensemble de fonctions de seuil. Si
:math:`X_i > s_i` alors il faut suivre cette branche, sinon, telle
autre. Les arbres de décision ne sont pas sensibles aux problèmes
d’échelle de variables. Si deux variables sont corrélées
:math:`cor(X_1, X_2)= 1`, l’arbre subit les mêmes problèmes qu’un modèle
linéaire. Dans le cas linéaire, il suffit de changer l’échelle
:math:`(X_1, \ln X_2)` pour éviter ce problème.


-  Pourquoi cette transformation ne change rien pour un arbre de
   décision ?
-  Quelle corrélation il faudrait calculer pour repérer les variables
   identiques selon le point de vue d’un arbre de décision ?