.. _discretgradientrst:

=========================
Le gradient et le discret
=========================


.. only:: html

    **Links:** :download:`notebook <discret_gradient.ipynb>`, :downloadlink:`html <discret_gradient2html.html>`, :download:`PDF <discret_gradient.pdf>`, :download:`python <discret_gradient.py>`, :downloadlink:`slides <discret_gradient.slides.html>`, :githublink:`GitHub|_doc/notebooks/dsgarden/discret_gradient.ipynb|*`


Les méthodes d’optimisation à base de gradient s’appuie sur une fonction
d’erreur dérivable qu’on devrait appliquer de préférence sur des
variables aléatoires réelles. Ce notebook explore quelques idées.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





Un petit problème simple
------------------------

On utilise le jeu de données *iris* disponible dans
`scikit-learn <http://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html>`__.

.. code:: ipython3

    from sklearn import datasets
    
    iris = datasets.load_iris()
    X = iris.data[:, :2]  # we only take the first two features.
    Y = iris.target

On cale une régression logistique. On ne distingue pas apprentissage et
test car ce n’est pas le propos de ce notebook.

.. code:: ipython3

    from sklearn.linear_model import LogisticRegression
    clf = LogisticRegression(multi_class="ovr", solver="liblinear")
    clf.fit(X, Y)




.. parsed-literal::
    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr',
              n_jobs=None, penalty='l2', random_state=None, solver='liblinear',
              tol=0.0001, verbose=0, warm_start=False)



Puis on calcule la matrice de confusion.

.. code:: ipython3

    from sklearn.metrics import confusion_matrix
    pred = clf.predict(X)
    confusion_matrix(Y, pred)




.. parsed-literal::
    array([[49,  1,  0],
           [ 2, 21, 27],
           [ 1,  4, 45]], dtype=int64)



Multiplication des observations
-------------------------------

Le paramètre ``multi_class='ovr'`` stipule que le modèle cache en fait
l’estimation de 3 régressions logistiques binaire. Essayons de n’en
faire qu’une seule en ajouter le label ``Y`` aux variables. Soit un
couple :math:`(X_i \in \mathbb{R^d}, Y_i \in \mathbb{N})` qui correspond
à une observation pour un problème multi-classe. Comme il y a :math:`C`
classes, on multiplie cette ligne par le nombre de classes :math:`C`
pour obtenir :

.. math:: \forall c \in \mathbb{[}1, ..., C\mathbb{]}, \; \left\{ \begin{array}{ll} X_i' = (X_{i,1}, ..., X_{i,d}, Y_{i,1}, ..., Y_{i,C}) \\ Y_i' = \mathbf{1\!\!1}_{Y_i = c} \\ Y_{i,k} = \mathbf{1\!\!1}_{c = k}\end{array} \right.

Voyons ce que cela donne sur un exemple :

.. code:: ipython3

    import numpy
    import pandas
    
    def multiplie(X, Y, classes=None):
        if classes is None:
            classes = numpy.unique(Y)
        XS = []
        YS = []
        for i in classes:
            X2 = numpy.zeros((X.shape[0], 3))
            X2[:,i] = 1
            Yb = Y == i
            XS.append(numpy.hstack([X, X2]))
            Yb = Yb.reshape((len(Yb), 1))
            YS.append(Yb)
    
        Xext = numpy.vstack(XS)
        Yext = numpy.vstack(YS)
        return Xext, Yext
    
    x, y = multiplie(X[:1,:], Y[:1], [0, 1, 2])
    df = pandas.DataFrame(numpy.hstack([x, y]))
    df.columns = ["X1", "X2", "Y0", "Y1", "Y2", "Y'"]
    df






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X1</th>
          <th>X2</th>
          <th>Y0</th>
          <th>Y1</th>
          <th>Y2</th>
          <th>Y'</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>1.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>0.0</td>
        </tr>
      </tbody>
    </table>
    </div>



Trois colonnes ont été ajoutées côté :math:`X`, la ligne a été
multipliée 3 fois, la dernière colonne est :math:`Y` qui ne vaut 1 que
lorsque le 1 est au bon endroit dans une des colonnes ajoutées. Le
problème de classification qui été de prédire la bonne classe devient :
est-ce la classe à prédire est :math:`k` ? On applique cela sur toutes
les lignes de la base et cela donne :

.. code:: ipython3

    Xext, Yext = multiplie(X, Y)
    numpy.hstack([Xext, Yext])
    df = pandas.DataFrame(numpy.hstack([Xext, Yext]))
    df.columns = ["X1", "X2", "Y0", "Y1", "Y2", "Y'"]
    df.iloc[numpy.random.permutation(df.index), :].head(n=10)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X1</th>
          <th>X2</th>
          <th>Y0</th>
          <th>Y1</th>
          <th>Y2</th>
          <th>Y'</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>414</th>
          <td>6.7</td>
          <td>3.3</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>125</th>
          <td>5.5</td>
          <td>2.5</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>394</th>
          <td>6.7</td>
          <td>3.1</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>411</th>
          <td>6.0</td>
          <td>2.2</td>
          <td>1.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>95</th>
          <td>7.6</td>
          <td>3.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>64</th>
          <td>5.8</td>
          <td>2.6</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>309</th>
          <td>5.0</td>
          <td>3.4</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>7</th>
          <td>6.7</td>
          <td>3.0</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>182</th>
          <td>6.1</td>
          <td>2.8</td>
          <td>1.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>49</th>
          <td>4.7</td>
          <td>3.2</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    from sklearn.ensemble import GradientBoostingClassifier
    clf = GradientBoostingClassifier()
    clf.fit(Xext, Yext.ravel())




.. parsed-literal::
    GradientBoostingClassifier(criterion='friedman_mse', init=None,
                  learning_rate=0.1, loss='deviance', max_depth=3,
                  max_features=None, max_leaf_nodes=None,
                  min_impurity_decrease=0.0, min_impurity_split=None,
                  min_samples_leaf=1, min_samples_split=2,
                  min_weight_fraction_leaf=0.0, n_estimators=100,
                  n_iter_no_change=None, presort='auto', random_state=None,
                  subsample=1.0, tol=0.0001, validation_fraction=0.1,
                  verbose=0, warm_start=False)



.. code:: ipython3

    pred = clf.predict(Xext)
    confusion_matrix(Yext, pred)




.. parsed-literal::
    array([[278,  22],
           [ 25, 125]], dtype=int64)



Introduire du bruit
-------------------

Un des problèmes de cette méthode est qu’on ajoute une variable binaire
pour un problème résolu à l’aide d’une optimisation à base de gradient.
C’est moyen. Pas de problème, changeons un peu la donne.

.. code:: ipython3

    def multiplie_bruit(X, Y, classes=None):
        if classes is None:
            classes = numpy.unique(Y)
        XS = []
        YS = []
        for i in classes:
            # X2 = numpy.random.randn((X.shape[0]* 3)).reshape(X.shape[0], 3) * 0.1
            X2 = numpy.random.random((X.shape[0], 3)) * 0.2
            X2[:,i] += 1
            Yb = Y == i
            XS.append(numpy.hstack([X, X2]))
            Yb = Yb.reshape((len(Yb), 1))
            YS.append(Yb)
    
        Xext = numpy.vstack(XS)
        Yext = numpy.vstack(YS)
        return Xext, Yext
    
    x, y = multiplie_bruit(X[:1,:], Y[:1], [0, 1, 2])
    df = pandas.DataFrame(numpy.hstack([x, y]))
    df.columns = ["X1", "X2", "Y0", "Y1", "Y2", "Y'"]
    df






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X1</th>
          <th>X2</th>
          <th>Y0</th>
          <th>Y1</th>
          <th>Y2</th>
          <th>Y'</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>1.107461</td>
          <td>0.166893</td>
          <td>0.018765</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>0.162464</td>
          <td>1.187359</td>
          <td>0.187721</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>0.086876</td>
          <td>0.178472</td>
          <td>1.179201</td>
          <td>0.0</td>
        </tr>
      </tbody>
    </table>
    </div>



Le problème est le même qu’avant excepté les variables :math:`Y_i` qui
sont maintenant réel. Au lieu d’être nul, on prend une valeur
:math:`Y_i < 0.4`.

.. code:: ipython3

    Xextb, Yextb = multiplie_bruit(X, Y)
    df = pandas.DataFrame(numpy.hstack([Xextb, Yextb]))
    df.columns = ["X1", "X2", "Y0", "Y1", "Y2", "Y'"]
    df.iloc[numpy.random.permutation(df.index), :].head(n=10)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X1</th>
          <th>X2</th>
          <th>Y0</th>
          <th>Y1</th>
          <th>Y2</th>
          <th>Y'</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>295</th>
          <td>5.5</td>
          <td>2.6</td>
          <td>0.197643</td>
          <td>1.199976</td>
          <td>0.180766</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>46</th>
          <td>5.2</td>
          <td>3.4</td>
          <td>0.178395</td>
          <td>0.190600</td>
          <td>1.159765</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>187</th>
          <td>6.7</td>
          <td>3.1</td>
          <td>0.188947</td>
          <td>1.093288</td>
          <td>0.139723</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>210</th>
          <td>6.9</td>
          <td>3.1</td>
          <td>0.095428</td>
          <td>0.182643</td>
          <td>1.037533</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>29</th>
          <td>5.5</td>
          <td>3.5</td>
          <td>1.131419</td>
          <td>0.077241</td>
          <td>0.177483</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>315</th>
          <td>6.4</td>
          <td>3.2</td>
          <td>0.099738</td>
          <td>0.197291</td>
          <td>1.035431</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>152</th>
          <td>5.8</td>
          <td>2.7</td>
          <td>0.069061</td>
          <td>0.045325</td>
          <td>1.061221</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>168</th>
          <td>6.5</td>
          <td>2.8</td>
          <td>0.093164</td>
          <td>1.177413</td>
          <td>0.095890</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>348</th>
          <td>6.9</td>
          <td>3.1</td>
          <td>1.094184</td>
          <td>0.196944</td>
          <td>0.083975</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>261</th>
          <td>6.3</td>
          <td>2.8</td>
          <td>0.197558</td>
          <td>0.080273</td>
          <td>1.009379</td>
          <td>1.0</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    from sklearn.ensemble import GradientBoostingClassifier
    clfb = GradientBoostingClassifier()
    clfb.fit(Xextb, Yextb.ravel())




.. parsed-literal::
    GradientBoostingClassifier(criterion='friedman_mse', init=None,
                  learning_rate=0.1, loss='deviance', max_depth=3,
                  max_features=None, max_leaf_nodes=None,
                  min_impurity_decrease=0.0, min_impurity_split=None,
                  min_samples_leaf=1, min_samples_split=2,
                  min_weight_fraction_leaf=0.0, n_estimators=100,
                  n_iter_no_change=None, presort='auto', random_state=None,
                  subsample=1.0, tol=0.0001, validation_fraction=0.1,
                  verbose=0, warm_start=False)



.. code:: ipython3

    predb = clfb.predict(Xextb)
    confusion_matrix(Yextb, predb)




.. parsed-literal::
    array([[299,   1],
           [ 10, 140]], dtype=int64)



C’est un petit peu mieux.

Comparaisons de plusieurs modèles
---------------------------------

On cherche maintenant à comparer le gain en introduisant du bruit pour
différents modèles.

.. code:: ipython3

    def error(model, x, y):
        p = model.predict(x)
        cm = confusion_matrix(y, p)
        return (cm[1,0] + cm[0,1]) / cm.sum()
    
    def comparaison(model, X, Y):
    
        if isinstance(model, tuple):
            clf = model[0](**model[1])
            clfb = model[0](**model[1])
            model = model[0]
        else:        
            clf = model()
            clfb = model()
        
        Xext, Yext = multiplie(X, Y)
        clf.fit(Xext, Yext.ravel())
        err = error(clf, Xext, Yext)
        
        Xextb, Yextb = multiplie_bruit(X, Y)
        clfb.fit(Xextb, Yextb.ravel())
        errb = error(clfb, Xextb, Yextb)
        return dict(model=model.__name__, err1=err, err2=errb)
    
    from sklearn.linear_model import LogisticRegression
    from sklearn.tree import DecisionTreeClassifier, ExtraTreeClassifier
    from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier, AdaBoostClassifier
    from sklearn.neural_network import MLPClassifier
    from sklearn.naive_bayes import GaussianNB
    from sklearn.neighbors import KNeighborsClassifier, RadiusNeighborsClassifier
    from xgboost import XGBClassifier
    
    models = [(LogisticRegression, dict(multi_class="ovr", solver="liblinear")),
              GradientBoostingClassifier,
              (RandomForestClassifier, dict(n_estimators=20)),
              DecisionTreeClassifier,
              ExtraTreeClassifier,
              XGBClassifier,
              (ExtraTreesClassifier, dict(n_estimators=20)),
              (MLPClassifier, dict(activation="logistic")),
              GaussianNB, KNeighborsClassifier, 
              (AdaBoostClassifier, dict(base_estimator=LogisticRegression(multi_class="ovr", solver="liblinear"), 
                                        algorithm="SAMME"))]
    
    res = [comparaison(model, X, Y) for model in models]
    df = pandas.DataFrame(res)
    df.sort_values("model")






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>err1</th>
          <th>err2</th>
          <th>model</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>10</th>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>AdaBoostClassifier</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0.048889</td>
          <td>0.000000</td>
          <td>DecisionTreeClassifier</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0.048889</td>
          <td>0.000000</td>
          <td>ExtraTreeClassifier</td>
        </tr>
        <tr>
          <th>6</th>
          <td>0.048889</td>
          <td>0.000000</td>
          <td>ExtraTreesClassifier</td>
        </tr>
        <tr>
          <th>8</th>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>GaussianNB</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.104444</td>
          <td>0.044444</td>
          <td>GradientBoostingClassifier</td>
        </tr>
        <tr>
          <th>9</th>
          <td>0.104444</td>
          <td>0.091111</td>
          <td>KNeighborsClassifier</td>
        </tr>
        <tr>
          <th>0</th>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>LogisticRegression</td>
        </tr>
        <tr>
          <th>7</th>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>MLPClassifier</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.053333</td>
          <td>0.002222</td>
          <td>RandomForestClassifier</td>
        </tr>
        <tr>
          <th>5</th>
          <td>0.333333</td>
          <td>0.053333</td>
          <td>XGBClassifier</td>
        </tr>
      </tbody>
    </table>
    </div>



*err1* correspond à :math:`Y_0, Y_1, Y_2` binaire, *err2* aux mêmes
variables mais avec un peu de bruit. L’ajout ne semble pas faire
décroître la performance et l’améliore dans certains cas. C’est une
piste à suivre. Reste à savoir si les modèles n’apprennent pas le bruit.

Avec une ACP
------------

On peut faire varier le nombre de composantes, j’en ai gardé qu’une.
L’ACP est appliquée après avoir ajouté les variables binaires ou
binaires bruitées. Le résultat est sans équivoque. Aucun modèle ne
parvient à apprendre sans l’ajout de bruit.

.. code:: ipython3

    from sklearn.decomposition import PCA
    
    def comparaison_ACP(model, X, Y):
    
        if isinstance(model, tuple):
            clf = model[0](**model[1])
            clfb = model[0](**model[1])
            model = model[0]
        else:        
            clf = model()
            clfb = model()
        
        axes = 1
        solver = "full"
        Xext, Yext = multiplie(X, Y)
        Xext = PCA(n_components=axes, svd_solver=solver).fit_transform(Xext)
        clf.fit(Xext, Yext.ravel())
        err = error(clf, Xext, Yext)
        
        Xextb, Yextb = multiplie_bruit(X, Y)
        Xextb = PCA(n_components=axes, svd_solver=solver).fit_transform(Xextb)
        clfb.fit(Xextb, Yextb.ravel())
        errb = error(clfb, Xextb, Yextb)
        return dict(modelACP=model.__name__, errACP1=err, errACP2=errb)
    
    res = [comparaison_ACP(model, X, Y) for model in models]
    dfb = pandas.DataFrame(res)
    pandas.concat([ df.sort_values("model"), dfb.sort_values("modelACP")], axis=1)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>err1</th>
          <th>err2</th>
          <th>model</th>
          <th>errACP1</th>
          <th>errACP2</th>
          <th>modelACP</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>10</th>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>AdaBoostClassifier</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>AdaBoostClassifier</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0.048889</td>
          <td>0.000000</td>
          <td>DecisionTreeClassifier</td>
          <td>0.333333</td>
          <td>0.000000</td>
          <td>DecisionTreeClassifier</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0.048889</td>
          <td>0.000000</td>
          <td>ExtraTreeClassifier</td>
          <td>0.333333</td>
          <td>0.000000</td>
          <td>ExtraTreeClassifier</td>
        </tr>
        <tr>
          <th>6</th>
          <td>0.048889</td>
          <td>0.000000</td>
          <td>ExtraTreesClassifier</td>
          <td>0.333333</td>
          <td>0.000000</td>
          <td>ExtraTreesClassifier</td>
        </tr>
        <tr>
          <th>8</th>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>GaussianNB</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>GaussianNB</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.104444</td>
          <td>0.044444</td>
          <td>GradientBoostingClassifier</td>
          <td>0.333333</td>
          <td>0.224444</td>
          <td>GradientBoostingClassifier</td>
        </tr>
        <tr>
          <th>9</th>
          <td>0.104444</td>
          <td>0.091111</td>
          <td>KNeighborsClassifier</td>
          <td>0.335556</td>
          <td>0.340000</td>
          <td>KNeighborsClassifier</td>
        </tr>
        <tr>
          <th>0</th>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>LogisticRegression</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>LogisticRegression</td>
        </tr>
        <tr>
          <th>7</th>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>MLPClassifier</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>MLPClassifier</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.053333</td>
          <td>0.002222</td>
          <td>RandomForestClassifier</td>
          <td>0.333333</td>
          <td>0.024444</td>
          <td>RandomForestClassifier</td>
        </tr>
        <tr>
          <th>5</th>
          <td>0.333333</td>
          <td>0.053333</td>
          <td>XGBClassifier</td>
          <td>0.333333</td>
          <td>0.315556</td>
          <td>XGBClassifier</td>
        </tr>
      </tbody>
    </table>
    </div>



Base d’apprentissage et de test
-------------------------------

Cette fois-ci, on s’intéresse à la qualité des frontières que les
modèles trouvent en vérifiant sur une base de test que l’apprentissage
s’est bien passé.

.. code:: ipython3

    from sklearn.model_selection import train_test_split
    
    def comparaison_train_test(models, X, Y, mbruit=multiplie_bruit, acp=None):
    
        axes = acp
        solver = "full"        
            
        ind = numpy.random.permutation(numpy.arange(X.shape[0]))
        X = X[ind,:]
        Y = Y[ind]
        X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=1./3)
        
        res = []
        for model in models:
        
            if isinstance(model, tuple):
                clf = model[0](**model[1])
                clfb = model[0](**model[1])
                model = model[0]
            else:        
                clf = model()
                clfb = model()
    
            Xext_train, Yext_train = multiplie(X_train, Y_train)
            Xext_test, Yext_test = multiplie(X_test, Y_test)
            if acp:
                Xext_train_ = Xext_train
                Xext_test_ = Xext_test
                acp_model = PCA(n_components=axes, svd_solver=solver).fit(Xext_train)
                Xext_train = acp_model.transform(Xext_train)
                Xext_test = acp_model.transform(Xext_test)        
            clf.fit(Xext_train, Yext_train.ravel())
    
            err_train = error(clf, Xext_train, Yext_train)
            err_test = error(clf, Xext_test, Yext_test)
    
            Xextb_train, Yextb_train = mbruit(X_train, Y_train)
            Xextb_test, Yextb_test = mbruit(X_test, Y_test)
            if acp:
                acp_model = PCA(n_components=axes, svd_solver=solver).fit(Xextb_train)
                Xextb_train = acp_model.transform(Xextb_train)
                Xextb_test = acp_model.transform(Xextb_test)        
                Xext_train = acp_model.transform(Xext_train_)
                Xext_test = acp_model.transform(Xext_test_)        
            clfb.fit(Xextb_train, Yextb_train.ravel())
    
            errb_train = error(clfb, Xextb_train, Yextb_train)
            errb_train_clean = error(clfb, Xext_train, Yext_train)
            errb_test = error(clfb, Xextb_test, Yextb_test)
            errb_test_clean = error(clfb, Xext_test, Yext_test)
            
            res.append(dict(modelTT=model.__name__, err_train=err_train, err2_train=errb_train,
                   err_test=err_test, err2_test=errb_test, err2b_test_clean=errb_test_clean,
                   err2b_train_clean=errb_train_clean))
            
        dfb = pandas.DataFrame(res)
        dfb = dfb[["modelTT", "err_train", "err2_train", "err2b_train_clean", "err_test", "err2_test", "err2b_test_clean"]]
        dfb = dfb.sort_values("modelTT")        
        return dfb
    
    dfb = comparaison_train_test(models, X, Y)
    dfb






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>modelTT</th>
          <th>err_train</th>
          <th>err2_train</th>
          <th>err2b_train_clean</th>
          <th>err_test</th>
          <th>err2_test</th>
          <th>err2b_test_clean</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>10</th>
          <td>AdaBoostClassifier</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>3</th>
          <td>DecisionTreeClassifier</td>
          <td>0.026667</td>
          <td>0.000000</td>
          <td>0.226667</td>
          <td>0.206667</td>
          <td>0.273333</td>
          <td>0.313333</td>
        </tr>
        <tr>
          <th>4</th>
          <td>ExtraTreeClassifier</td>
          <td>0.026667</td>
          <td>0.000000</td>
          <td>0.253333</td>
          <td>0.213333</td>
          <td>0.253333</td>
          <td>0.273333</td>
        </tr>
        <tr>
          <th>6</th>
          <td>ExtraTreesClassifier</td>
          <td>0.026667</td>
          <td>0.000000</td>
          <td>0.140000</td>
          <td>0.200000</td>
          <td>0.213333</td>
          <td>0.220000</td>
        </tr>
        <tr>
          <th>8</th>
          <td>GaussianNB</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>1</th>
          <td>GradientBoostingClassifier</td>
          <td>0.080000</td>
          <td>0.013333</td>
          <td>0.176667</td>
          <td>0.186667</td>
          <td>0.246667</td>
          <td>0.240000</td>
        </tr>
        <tr>
          <th>9</th>
          <td>KNeighborsClassifier</td>
          <td>0.070000</td>
          <td>0.076667</td>
          <td>0.073333</td>
          <td>0.160000</td>
          <td>0.160000</td>
          <td>0.166667</td>
        </tr>
        <tr>
          <th>0</th>
          <td>LogisticRegression</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>7</th>
          <td>MLPClassifier</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>2</th>
          <td>RandomForestClassifier</td>
          <td>0.026667</td>
          <td>0.000000</td>
          <td>0.156667</td>
          <td>0.206667</td>
          <td>0.266667</td>
          <td>0.213333</td>
        </tr>
        <tr>
          <th>5</th>
          <td>XGBClassifier</td>
          <td>0.106667</td>
          <td>0.036667</td>
          <td>0.333333</td>
          <td>0.193333</td>
          <td>0.280000</td>
          <td>0.346667</td>
        </tr>
      </tbody>
    </table>
    </div>



Les colonnes *err2b_train_clean* et *err2b_test_clean* sont les erreurs
obtenues par des modèles appris sur des colonnes bruitées et testées sur
des colonnes non bruitées ce qui est le véritable test. On s’aperçoit
que les performances sont très dégradées sur la base d’test. Une raison
est que le bruit choisi ajouté n’est pas centré. Corrigeons cela.

.. code:: ipython3

    def multiplie_bruit_centree(X, Y, classes=None):
        if classes is None:
            classes = numpy.unique(Y)
        XS = []
        YS = []
        for i in classes:
            # X2 = numpy.random.randn((X.shape[0]* 3)).reshape(X.shape[0], 3) * 0.1
            X2 = numpy.random.random((X.shape[0], 3)) * 0.2 - 0.1
            X2[:,i] += 1
            Yb = Y == i
            XS.append(numpy.hstack([X, X2]))
            Yb = Yb.reshape((len(Yb), 1))
            YS.append(Yb)
    
        Xext = numpy.vstack(XS)
        Yext = numpy.vstack(YS)
        return Xext, Yext
    
    dfb = comparaison_train_test(models, X, Y, mbruit=multiplie_bruit_centree, acp=None)
    dfb






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>modelTT</th>
          <th>err_train</th>
          <th>err2_train</th>
          <th>err2b_train_clean</th>
          <th>err_test</th>
          <th>err2_test</th>
          <th>err2b_test_clean</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>10</th>
          <td>AdaBoostClassifier</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>3</th>
          <td>DecisionTreeClassifier</td>
          <td>0.033333</td>
          <td>0.000000</td>
          <td>0.143333</td>
          <td>0.193333</td>
          <td>0.273333</td>
          <td>0.206667</td>
        </tr>
        <tr>
          <th>4</th>
          <td>ExtraTreeClassifier</td>
          <td>0.033333</td>
          <td>0.000000</td>
          <td>0.143333</td>
          <td>0.226667</td>
          <td>0.233333</td>
          <td>0.180000</td>
        </tr>
        <tr>
          <th>6</th>
          <td>ExtraTreesClassifier</td>
          <td>0.033333</td>
          <td>0.000000</td>
          <td>0.123333</td>
          <td>0.200000</td>
          <td>0.213333</td>
          <td>0.193333</td>
        </tr>
        <tr>
          <th>8</th>
          <td>GaussianNB</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>1</th>
          <td>GradientBoostingClassifier</td>
          <td>0.083333</td>
          <td>0.013333</td>
          <td>0.203333</td>
          <td>0.193333</td>
          <td>0.226667</td>
          <td>0.280000</td>
        </tr>
        <tr>
          <th>9</th>
          <td>KNeighborsClassifier</td>
          <td>0.106667</td>
          <td>0.106667</td>
          <td>0.100000</td>
          <td>0.180000</td>
          <td>0.180000</td>
          <td>0.193333</td>
        </tr>
        <tr>
          <th>0</th>
          <td>LogisticRegression</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>7</th>
          <td>MLPClassifier</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>2</th>
          <td>RandomForestClassifier</td>
          <td>0.040000</td>
          <td>0.000000</td>
          <td>0.176667</td>
          <td>0.206667</td>
          <td>0.240000</td>
          <td>0.253333</td>
        </tr>
        <tr>
          <th>5</th>
          <td>XGBClassifier</td>
          <td>0.080000</td>
          <td>0.063333</td>
          <td>0.170000</td>
          <td>0.186667</td>
          <td>0.220000</td>
          <td>0.240000</td>
        </tr>
      </tbody>
    </table>
    </div>



C’est mieux mais on en conclut que dans la plupart des cas, la meilleure
performance sur la base d’apprentissage avec le bruit ajouté est due au
fait que les modèles apprennent par coeur. Sur la base de test, les
performances ne sont pas meilleures. Une erreur de 33% signifie que la
réponse du classifieur est constante. On multiplie les exemples.

.. code:: ipython3

    def multiplie_bruit_centree_duplique(X, Y, classes=None):
        if classes is None:
            classes = numpy.unique(Y)
        XS = []
        YS = []
        for i in classes:
            
            for k in range(0,5):
                #X2 = numpy.random.randn((X.shape[0]* 3)).reshape(X.shape[0], 3) * 0.3
                X2 = numpy.random.random((X.shape[0], 3)) * 0.8 - 0.4
                X2[:,i] += 1
                Yb = Y == i
                XS.append(numpy.hstack([X, X2]))
                Yb = Yb.reshape((len(Yb), 1))
                YS.append(Yb)
                    
        Xext = numpy.vstack(XS)
        Yext = numpy.vstack(YS)
        return Xext, Yext
    
    dfb = comparaison_train_test(models, X, Y, mbruit=multiplie_bruit_centree_duplique, acp=None)
    dfb






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>modelTT</th>
          <th>err_train</th>
          <th>err2_train</th>
          <th>err2b_train_clean</th>
          <th>err_test</th>
          <th>err2_test</th>
          <th>err2b_test_clean</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>10</th>
          <td>AdaBoostClassifier</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>3</th>
          <td>DecisionTreeClassifier</td>
          <td>0.040000</td>
          <td>0.000000</td>
          <td>0.120000</td>
          <td>0.180000</td>
          <td>0.209333</td>
          <td>0.240000</td>
        </tr>
        <tr>
          <th>4</th>
          <td>ExtraTreeClassifier</td>
          <td>0.040000</td>
          <td>0.000000</td>
          <td>0.073333</td>
          <td>0.213333</td>
          <td>0.232000</td>
          <td>0.220000</td>
        </tr>
        <tr>
          <th>6</th>
          <td>ExtraTreesClassifier</td>
          <td>0.040000</td>
          <td>0.000000</td>
          <td>0.066667</td>
          <td>0.213333</td>
          <td>0.168000</td>
          <td>0.160000</td>
        </tr>
        <tr>
          <th>8</th>
          <td>GaussianNB</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>1</th>
          <td>GradientBoostingClassifier</td>
          <td>0.086667</td>
          <td>0.087333</td>
          <td>0.166667</td>
          <td>0.173333</td>
          <td>0.192000</td>
          <td>0.186667</td>
        </tr>
        <tr>
          <th>9</th>
          <td>KNeighborsClassifier</td>
          <td>0.110000</td>
          <td>0.094667</td>
          <td>0.106667</td>
          <td>0.113333</td>
          <td>0.158667</td>
          <td>0.153333</td>
        </tr>
        <tr>
          <th>0</th>
          <td>LogisticRegression</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>7</th>
          <td>MLPClassifier</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>2</th>
          <td>RandomForestClassifier</td>
          <td>0.046667</td>
          <td>0.000667</td>
          <td>0.090000</td>
          <td>0.160000</td>
          <td>0.188000</td>
          <td>0.226667</td>
        </tr>
        <tr>
          <th>5</th>
          <td>XGBClassifier</td>
          <td>0.123333</td>
          <td>0.108667</td>
          <td>0.173333</td>
          <td>0.153333</td>
          <td>0.204000</td>
          <td>0.193333</td>
        </tr>
      </tbody>
    </table>
    </div>



Cela fonctionne un peu mieux le fait d’ajouter du hasard ne permet pas
d’obtenir des gains significatifs à part pour le modèle
`SVC <http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html>`__.

.. code:: ipython3

    def multiplie_bruit_centree_duplique_rebalance(X, Y, classes=None):
        if classes is None:
            classes = numpy.unique(Y)
        XS = []
        YS = []
        for i in classes:
            
            X2 = numpy.random.random((X.shape[0], 3)) * 0.8 - 0.4
            X2[:,i] += 1  # * ((i % 2) * 2 - 1)
            Yb = Y == i
            XS.append(numpy.hstack([X, X2]))
            Yb = Yb.reshape((len(Yb), 1))
            YS.append(Yb)
                                      
                    
        Xext = numpy.vstack(XS)
        Yext = numpy.vstack(YS)
        return Xext, Yext
    
    dfb = comparaison_train_test(models, X, Y, mbruit=multiplie_bruit_centree_duplique_rebalance)
    dfb






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>modelTT</th>
          <th>err_train</th>
          <th>err2_train</th>
          <th>err2b_train_clean</th>
          <th>err_test</th>
          <th>err2_test</th>
          <th>err2b_test_clean</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>10</th>
          <td>AdaBoostClassifier</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>3</th>
          <td>DecisionTreeClassifier</td>
          <td>0.033333</td>
          <td>0.000000</td>
          <td>0.143333</td>
          <td>0.200000</td>
          <td>0.233333</td>
          <td>0.193333</td>
        </tr>
        <tr>
          <th>4</th>
          <td>ExtraTreeClassifier</td>
          <td>0.033333</td>
          <td>0.000000</td>
          <td>0.246667</td>
          <td>0.233333</td>
          <td>0.320000</td>
          <td>0.300000</td>
        </tr>
        <tr>
          <th>6</th>
          <td>ExtraTreesClassifier</td>
          <td>0.033333</td>
          <td>0.000000</td>
          <td>0.143333</td>
          <td>0.206667</td>
          <td>0.220000</td>
          <td>0.180000</td>
        </tr>
        <tr>
          <th>8</th>
          <td>GaussianNB</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>1</th>
          <td>GradientBoostingClassifier</td>
          <td>0.090000</td>
          <td>0.013333</td>
          <td>0.133333</td>
          <td>0.220000</td>
          <td>0.206667</td>
          <td>0.186667</td>
        </tr>
        <tr>
          <th>9</th>
          <td>KNeighborsClassifier</td>
          <td>0.103333</td>
          <td>0.110000</td>
          <td>0.123333</td>
          <td>0.206667</td>
          <td>0.180000</td>
          <td>0.186667</td>
        </tr>
        <tr>
          <th>0</th>
          <td>LogisticRegression</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>7</th>
          <td>MLPClassifier</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
          <td>0.333333</td>
        </tr>
        <tr>
          <th>2</th>
          <td>RandomForestClassifier</td>
          <td>0.040000</td>
          <td>0.000000</td>
          <td>0.146667</td>
          <td>0.180000</td>
          <td>0.266667</td>
          <td>0.173333</td>
        </tr>
        <tr>
          <th>5</th>
          <td>XGBClassifier</td>
          <td>0.100000</td>
          <td>0.033333</td>
          <td>0.210000</td>
          <td>0.206667</td>
          <td>0.240000</td>
          <td>0.246667</td>
        </tr>
      </tbody>
    </table>
    </div>



Petite explication
------------------

Dans tout le notebook, le score de la régression logistique est nul.
Elle ne parvient pas à apprendre tout simplement parce que le problème
choisi n’est pas linéaire séparable. S’il l’était, cela voudrait dire
que le problème suivant l’est aussi.

.. code:: ipython3

    M = numpy.zeros((9, 6))
    Y = numpy.zeros((9, 1))
    for i in range(0, 9):
        M[i, i//3] = 1
        M[i, i%3+3] = 1
        Y[i] = 1 if i//3 == i%3 else 0
    M,Y




.. parsed-literal::
    (array([[1., 0., 0., 1., 0., 0.],
            [1., 0., 0., 0., 1., 0.],
            [1., 0., 0., 0., 0., 1.],
            [0., 1., 0., 1., 0., 0.],
            [0., 1., 0., 0., 1., 0.],
            [0., 1., 0., 0., 0., 1.],
            [0., 0., 1., 1., 0., 0.],
            [0., 0., 1., 0., 1., 0.],
            [0., 0., 1., 0., 0., 1.]]), array([[1.],
            [0.],
            [0.],
            [0.],
            [1.],
            [0.],
            [0.],
            [0.],
            [1.]]))



.. code:: ipython3

    clf = LogisticRegression(multi_class="ovr", solver="liblinear")
    clf.fit(M, Y.ravel())




.. parsed-literal::
    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr',
              n_jobs=None, penalty='l2', random_state=None, solver='liblinear',
              tol=0.0001, verbose=0, warm_start=False)



.. code:: ipython3

    clf.predict(M)




.. parsed-literal::
    array([0., 0., 0., 0., 0., 0., 0., 0., 0.])



A revisiter.