.. _td2acorrectionclreganomalyrst:

============================================================
2A.data - Classification, régression, anomalies - correction
============================================================


.. only:: html

    **Links:** :download:`notebook <td2a_correction_cl_reg_anomaly.ipynb>`, :downloadlink:`html <td2a_correction_cl_reg_anomaly2html.html>`, :download:`python <td2a_correction_cl_reg_anomaly.py>`, :downloadlink:`slides <td2a_correction_cl_reg_anomaly.slides.html>`, :githublink:`GitHub|_doc/notebooks/td2a_ml/td2a_correction_cl_reg_anomaly.ipynb|*`


Le jeu de données `Wine Quality Data
Set <https://archive.ics.uci.edu/ml/datasets/Wine+Quality>`__ contient
5000 vins décrits par leurs caractéristiques chimiques et évalués par un
expert. Peut-on s’approcher de l’expert à l’aide d’un modèle de machine
learning.

.. code:: ipython3

    %matplotlib inline
    import matplotlib.pyplot as plt

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





Les données
-----------

On peut les récupérer sur
`github…data_2a <https://github.com/sdpython/ensae_teaching_cs/tree/master/src/ensae_teaching_cs/data/data_1a>`__.

.. code:: ipython3

    from ensae_teaching_cs.data import wines_quality
    from pandas import read_csv
    df = read_csv(wines_quality(local=True, filename=True))
    df.head()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>fixed_acidity</th>
          <th>volatile_acidity</th>
          <th>citric_acid</th>
          <th>residual_sugar</th>
          <th>chlorides</th>
          <th>free_sulfur_dioxide</th>
          <th>total_sulfur_dioxide</th>
          <th>density</th>
          <th>pH</th>
          <th>sulphates</th>
          <th>alcohol</th>
          <th>quality</th>
          <th>color</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>7.4</td>
          <td>0.70</td>
          <td>0.00</td>
          <td>1.9</td>
          <td>0.076</td>
          <td>11.0</td>
          <td>34.0</td>
          <td>0.9978</td>
          <td>3.51</td>
          <td>0.56</td>
          <td>9.4</td>
          <td>5</td>
          <td>red</td>
        </tr>
        <tr>
          <th>1</th>
          <td>7.8</td>
          <td>0.88</td>
          <td>0.00</td>
          <td>2.6</td>
          <td>0.098</td>
          <td>25.0</td>
          <td>67.0</td>
          <td>0.9968</td>
          <td>3.20</td>
          <td>0.68</td>
          <td>9.8</td>
          <td>5</td>
          <td>red</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7.8</td>
          <td>0.76</td>
          <td>0.04</td>
          <td>2.3</td>
          <td>0.092</td>
          <td>15.0</td>
          <td>54.0</td>
          <td>0.9970</td>
          <td>3.26</td>
          <td>0.65</td>
          <td>9.8</td>
          <td>5</td>
          <td>red</td>
        </tr>
        <tr>
          <th>3</th>
          <td>11.2</td>
          <td>0.28</td>
          <td>0.56</td>
          <td>1.9</td>
          <td>0.075</td>
          <td>17.0</td>
          <td>60.0</td>
          <td>0.9980</td>
          <td>3.16</td>
          <td>0.58</td>
          <td>9.8</td>
          <td>6</td>
          <td>red</td>
        </tr>
        <tr>
          <th>4</th>
          <td>7.4</td>
          <td>0.70</td>
          <td>0.00</td>
          <td>1.9</td>
          <td>0.076</td>
          <td>11.0</td>
          <td>34.0</td>
          <td>0.9978</td>
          <td>3.51</td>
          <td>0.56</td>
          <td>9.4</td>
          <td>5</td>
          <td>red</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    ax = df['quality'].hist(bins=16)
    ax.set_title("Distribution des notes");



.. image:: td2a_correction_cl_reg_anomaly_5_0.png


Il y a peu de très mauvais ou très bons vins. On découpe en
apprentissage / test ce qui va nécessairement rendre leur prédiction
complexe : un modèle reproduit en quelque sorte ce qu’il voit.

.. code:: ipython3

    from sklearn.model_selection import train_test_split
    X = df.drop("quality", axis=1)
    y = df["quality"]
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)
    X_train.shape, X_test.shape




.. parsed-literal::
    ((3248, 12), (3249, 12))



.. code:: ipython3

    from pandas import DataFrame
    
    def distribution(y_train, y_test):
        df_train = DataFrame(dict(color=y_train))
        df_test = DataFrame(dict(color=y_test))
        df_train['ctrain'] = 1
        df_test['ctest'] = 1
        h_train = df_train.groupby('color').count()
        h_test  = df_test.groupby('color').count()
        merge = h_train.join(h_test, how='outer')
        merge["ratio"] = merge.ctest / merge.ctrain
        return merge
        
    distribution(y_train, y_test)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>ctrain</th>
          <th>ctest</th>
          <th>ratio</th>
        </tr>
        <tr>
          <th>color</th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>3</th>
          <td>12</td>
          <td>18</td>
          <td>1.500000</td>
        </tr>
        <tr>
          <th>4</th>
          <td>94</td>
          <td>122</td>
          <td>1.297872</td>
        </tr>
        <tr>
          <th>5</th>
          <td>1076</td>
          <td>1062</td>
          <td>0.986989</td>
        </tr>
        <tr>
          <th>6</th>
          <td>1429</td>
          <td>1407</td>
          <td>0.984605</td>
        </tr>
        <tr>
          <th>7</th>
          <td>536</td>
          <td>543</td>
          <td>1.013060</td>
        </tr>
        <tr>
          <th>8</th>
          <td>98</td>
          <td>95</td>
          <td>0.969388</td>
        </tr>
        <tr>
          <th>9</th>
          <td>3</td>
          <td>2</td>
          <td>0.666667</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    ax = y_train.hist(bins=24, label="train", align="right")
    y_test.hist(bins=24, label="test", ax=ax, align="left")
    ax.set_title("Distribution des notes")
    ax.legend();



.. image:: td2a_correction_cl_reg_anomaly_9_0.png


Avec un peu de chance, les notes extrêmes sont présentes dans les bases
d’apprentissages et de tests mais une note seule a peu d’influence sur
un modèle. Pour s’assurer une meilleur répartition train / test, on peut
s’assurer que chaque note est bien répartie de chaque côté. On se sert
du paramètre
`stratify <https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html>`__.

.. code:: ipython3

    X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.5)
    X_train.shape, X_test.shape
    ax = y_train.hist(bins=24, label="train", align="right")
    y_test.hist(bins=24, label="test", ax=ax, align="left")
    ax.set_title("Distribution des notes - statifiée")
    ax.legend();



.. image:: td2a_correction_cl_reg_anomaly_11_0.png


.. code:: ipython3

    distribution(y_train, y_test)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>ctrain</th>
          <th>ctest</th>
          <th>ratio</th>
        </tr>
        <tr>
          <th>color</th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>3</th>
          <td>15</td>
          <td>15</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>4</th>
          <td>108</td>
          <td>108</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>5</th>
          <td>1069</td>
          <td>1069</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>6</th>
          <td>1418</td>
          <td>1418</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>7</th>
          <td>539</td>
          <td>540</td>
          <td>1.001855</td>
        </tr>
        <tr>
          <th>8</th>
          <td>96</td>
          <td>97</td>
          <td>1.010417</td>
        </tr>
        <tr>
          <th>9</th>
          <td>3</td>
          <td>2</td>
          <td>0.666667</td>
        </tr>
      </tbody>
    </table>
    </div>



La répartition des notes selon apprentissage / test est plus uniforme.

Premier modèle
--------------

.. code:: ipython3

    from sklearn.linear_model import LogisticRegression
    logreg = LogisticRegression()
    try:
        logreg.fit(X_train, y_train)
    except Exception as e:
        print(e)


.. parsed-literal::
    could not convert string to float: 'white'


Une colonne n’est pas numérique. On utilise un
`OneHotEncoder <https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html>`__.

.. code:: ipython3

    from sklearn.preprocessing import OneHotEncoder
    one = OneHotEncoder()
    one.fit(X_train[['color']])
    tr = one.transform(X_test[["color"]])
    tr




.. parsed-literal::
    <3249x2 sparse matrix of type '<class 'numpy.float64'>'
    	with 3249 stored elements in Compressed Sparse Row format>



La matrice est `sparse ou
creuse <https://fr.wikipedia.org/wiki/Matrice_creuse>`__.

.. code:: ipython3

    tr.todense()[:5]




.. parsed-literal::
    matrix([[0., 1.],
            [0., 1.],
            [0., 1.],
            [0., 1.],
            [0., 1.]])



Ensuite il faut fusionner ces deux colonnes avec les données ou une
seule puisqu’elles sont corrélées. Ou alors on écrit un pipeline…

.. code:: ipython3

    from sklearn.pipeline import Pipeline
    from sklearn.compose import ColumnTransformer
    
    numeric_features = [c for c in X_train if c != 'color']
    
    pipe = Pipeline([
        ("prep", ColumnTransformer([
            ("color", Pipeline([
                ('one', OneHotEncoder()),
                ('select', ColumnTransformer([('sel1', 'passthrough', [0])]))
            ]), ['color']),
            ("others", "passthrough", numeric_features)
        ])),
    ])
    
    pipe.fit(X_train)
    pipe.transform(X_test)[:2]




.. parsed-literal::
    array([[0.0000e+00, 6.0000e+00, 3.4000e-01, 2.4000e-01, 5.4000e+00,
            6.0000e-02, 2.3000e+01, 1.2600e+02, 9.9510e-01, 3.2500e+00,
            4.4000e-01, 9.0000e+00],
           [0.0000e+00, 6.7000e+00, 4.1000e-01, 2.7000e-01, 2.6000e+00,
            3.3000e-02, 2.5000e+01, 8.5000e+01, 9.9086e-01, 3.0500e+00,
            3.4000e-01, 1.1700e+01]])



.. code:: ipython3

    from jyquickhelper import RenderJsDot
    from mlinsights.plotting import pipeline2dot
    dot = pipeline2dot(pipe, X_train)
    RenderJsDot(dot)






.. raw:: html

    <div id="M7e90b998101948bdaf61f9b4664354bf-cont"><div id="M7e90b998101948bdaf61f9b4664354bf" style="width:100%;height:100%;"></div></div>
    <script>

    require(['http://www.xavierdupre.fr/js/vizjs/viz.js'], function() { var svgGraph = Viz("digraph{\n  orientation=portrait;\n  nodesep=0.05;\n  ranksep=0.25;\n  sch0[label=\"<f0> fixed_acidity|<f1> volatile_acidity|<f2> citric_acid|<f3> residual_sugar|<f4> chlorides|<f5> free_sulfur_dioxide|<f6> total_sulfur_dioxide|<f7> density|<f8> pH|<f9> sulphates|<f10> alcohol|<f11> color\",shape=record,fontsize=8];\n\n  node1[label=\"OneHotEncoder\",shape=box,style=\"filled,rounded\",color=cyan,fontsize=12];\n  sch0:f11 -> node1;\n  sch1[label=\"<f0> color\",shape=record,fontsize=8];\n  node1 -> sch1:f0;\n\n  node2[label=\"Identity\",shape=box,style=\"filled,rounded\",color=cyan,fontsize=12];\n  sch0:f11 -> node2;\n  sch2[label=\"<f0> -|<f1> v|<f2> -|<f3> 0\",shape=record,fontsize=8];\n  node2 -> sch2:f2;\n  node2 -> sch2:f1;\n  node2 -> sch2:f3;\n\n  node3[label=\"Identity\",shape=box,style=\"filled,rounded\",color=cyan,fontsize=12];\n  sch0:f0 -> node3;\n  sch0:f1 -> node3;\n  sch0:f2 -> node3;\n  sch0:f3 -> node3;\n  sch0:f4 -> node3;\n  sch0:f5 -> node3;\n  sch0:f6 -> node3;\n  sch0:f7 -> node3;\n  sch0:f8 -> node3;\n  sch0:f9 -> node3;\n  sch0:f10 -> node3;\n  sch3[label=\"<f0> -v-1|<f1> -v-2|<f2> -v-3|<f3> -v-4|<f4> -v-5|<f5> -v-6|<f6> -v-7|<f7> -v-8|<f8> -v-9|<f9> -v-10|<f10> -v-11\",shape=record,fontsize=8];\n  node3 -> sch3:f0;\n  node3 -> sch3:f1;\n  node3 -> sch3:f2;\n  node3 -> sch3:f3;\n  node3 -> sch3:f4;\n  node3 -> sch3:f5;\n  node3 -> sch3:f6;\n  node3 -> sch3:f7;\n  node3 -> sch3:f8;\n  node3 -> sch3:f9;\n  node3 -> sch3:f10;\n\n  node4[label=\"union\",shape=box,style=\"filled,rounded\",color=cyan,fontsize=12];\n  sch2:f2 -> node4;\n  sch2:f1 -> node4;\n  sch2:f2 -> node4;\n  sch2:f3 -> node4;\n  sch3:f0 -> node4;\n  sch3:f1 -> node4;\n  sch3:f2 -> node4;\n  sch3:f3 -> node4;\n  sch3:f4 -> node4;\n  sch3:f5 -> node4;\n  sch3:f6 -> node4;\n  sch3:f7 -> node4;\n  sch3:f8 -> node4;\n  sch3:f9 -> node4;\n  sch3:f10 -> node4;\n  sch4[label=\"<f0> -v-12\",shape=record,fontsize=8];\n  node4 -> sch4:f0;\n}");
    document.getElementById('M7e90b998101948bdaf61f9b4664354bf').innerHTML = svgGraph; });

    </script>



Il reste quelques bugs. On ajoute un classifieur.

.. code:: ipython3

    pipe = Pipeline([
        ("prep", ColumnTransformer([
            ("color", Pipeline([
                ('one', OneHotEncoder()),
                ('select', ColumnTransformer([('sel1', 'passthrough', [0])]))
            ]), ['color']),
            ("others", "passthrough", numeric_features)
        ])),
        ("lr", LogisticRegression(max_iter=1000)),
    ])
    
    pipe.fit(X_train, y_train)


.. parsed-literal::
    C:\Python395_x64\lib\site-packages\sklearn\linear_model\_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      n_iter_i = _check_optimize_result(




.. parsed-literal::

    Pipeline(steps=[('prep',
                     ColumnTransformer(transformers=[('color',
                                                      Pipeline(steps=[('one',
                                                                       OneHotEncoder()),
                                                                      ('select',
                                                                       ColumnTransformer(transformers=[('sel1',
                                                                                                        'passthrough',
                                                                                                        [0])]))]),
                                                      ['color']),
                                                     ('others', 'passthrough',
                                                      ['fixed_acidity',
                                                       'volatile_acidity',
                                                       'citric_acid',
                                                       'residual_sugar',
                                                       'chlorides',
                                                       'free_sulfur_dioxide',
                                                       'total_sulfur_dioxide',
                                                       'density', 'pH', 'sulphates',
                                                       'alcohol'])])),
                    ('lr', LogisticRegression(max_iter=1000))])



.. code:: ipython3

    from sklearn.metrics import classification_report
    print(classification_report(y_test, pipe.predict(X_test)))


.. parsed-literal::
                  precision    recall  f1-score   support
               3       0.00      0.00      0.00        15
               4       0.00      0.00      0.00       108
               5       0.60      0.61      0.60      1069
               6       0.52      0.72      0.60      1418
               7       0.40      0.14      0.21       540
               8       0.00      0.00      0.00        97
               9       0.00      0.00      0.00         2
        accuracy                           0.54      3249
       macro avg       0.22      0.21      0.20      3249
    weighted avg       0.49      0.54      0.50      3249
    


.. parsed-literal::

    C:\Python395_x64\lib\site-packages\sklearn\metrics\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
      _warn_prf(average, modifier, msg_start, len(result))
    C:\Python395_x64\lib\site-packages\sklearn\metrics\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
      _warn_prf(average, modifier, msg_start, len(result))
    C:\Python395_x64\lib\site-packages\sklearn\metrics\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
      _warn_prf(average, modifier, msg_start, len(result))


Pas extraordinaire.

.. code:: ipython3

    from sklearn.ensemble import RandomForestClassifier
    
    pipe = Pipeline([
        ("prep", ColumnTransformer([
            ("color", Pipeline([
                ('one', OneHotEncoder()),
                ('select', ColumnTransformer([('sel1', 'passthrough', [0])]))
            ]), ['color']),
            ("others", "passthrough", numeric_features)
        ])),
        ("lr", RandomForestClassifier()),
    ])
    
    pipe.fit(X_train, y_train)
    print(classification_report(y_test, pipe.predict(X_test)))


.. parsed-literal::
                  precision    recall  f1-score   support
               3       0.00      0.00      0.00        15
               4       0.73      0.07      0.13       108
               5       0.70      0.71      0.70      1069
               6       0.62      0.76      0.68      1418
               7       0.65      0.44      0.52       540
               8       0.83      0.30      0.44        97
               9       0.00      0.00      0.00         2
        accuracy                           0.65      3249
       macro avg       0.50      0.33      0.35      3249
    weighted avg       0.66      0.65      0.63      3249
    


.. parsed-literal::

    C:\Python395_x64\lib\site-packages\sklearn\metrics\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
      _warn_prf(average, modifier, msg_start, len(result))
    C:\Python395_x64\lib\site-packages\sklearn\metrics\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
      _warn_prf(average, modifier, msg_start, len(result))
    C:\Python395_x64\lib\site-packages\sklearn\metrics\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
      _warn_prf(average, modifier, msg_start, len(result))


Beaucoup mieux.

Courbe ROC pour chaque classe
-----------------------------

.. code:: ipython3

    from sklearn.metrics import roc_curve, auc
    
    labels = pipe.steps[1][1].classes_
    y_score = pipe.predict_proba(X_test)
    
    fpr = dict()
    tpr = dict()
    roc_auc = dict()
    for i, cl in enumerate(labels):
        fpr[cl], tpr[cl], _ = roc_curve(y_test == cl, y_score[:, i])
        roc_auc[cl] = auc(fpr[cl], tpr[cl])

.. code:: ipython3

    fig, ax = plt.subplots(1, 1, figsize=(8,4))
    for k in roc_auc:
        ax.plot(fpr[k], tpr[k], label="c%d = %1.2f" % (k, roc_auc[k]))
    ax.legend();



.. image:: td2a_correction_cl_reg_anomaly_31_0.png


Ces chiffres peuvent paraître élevés mais ce n’est pas formidable quand
même.

.. code:: ipython3

    from sklearn.metrics import confusion_matrix
    confusion_matrix(y_test, pipe.predict(X_test), labels=labels)




.. parsed-literal::
    array([[   0,    0,    3,   12,    0,    0,    0],
           [   0,    8,   60,   38,    2,    0,    0],
           [   2,    3,  755,  303,    6,    0,    0],
           [   0,    0,  242, 1083,   91,    2,    0],
           [   0,    0,   20,  280,  236,    4,    0],
           [   0,    0,    0,   42,   26,   29,    0],
           [   0,    0,    0,    2,    0,    0,    0]], dtype=int64)



Ce n’est pas très joli…

.. code:: ipython3

    def confusion_matrix_df(y_test, y_true):
        conf = confusion_matrix(y_test, y_true)
        labels = list(sorted(set(y_test)))
        df = DataFrame(conf, columns=labels)
        df.set_index(labels)
        return df
    
    confusion_matrix_df(y_test, pipe.predict(X_test))






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>3</th>
          <th>4</th>
          <th>5</th>
          <th>6</th>
          <th>7</th>
          <th>8</th>
          <th>9</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0</td>
          <td>0</td>
          <td>3</td>
          <td>12</td>
          <td>0</td>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0</td>
          <td>8</td>
          <td>60</td>
          <td>38</td>
          <td>2</td>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>2</td>
          <td>3</td>
          <td>755</td>
          <td>303</td>
          <td>6</td>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0</td>
          <td>0</td>
          <td>242</td>
          <td>1083</td>
          <td>91</td>
          <td>2</td>
          <td>0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0</td>
          <td>0</td>
          <td>20</td>
          <td>280</td>
          <td>236</td>
          <td>4</td>
          <td>0</td>
        </tr>
        <tr>
          <th>5</th>
          <td>0</td>
          <td>0</td>
          <td>0</td>
          <td>42</td>
          <td>26</td>
          <td>29</td>
          <td>0</td>
        </tr>
        <tr>
          <th>6</th>
          <td>0</td>
          <td>0</td>
          <td>0</td>
          <td>2</td>
          <td>0</td>
          <td>0</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>



Mais cela veut dire que pour un score élevé, le taux de bonne
classification s’améliore.

.. code:: ipython3

    import numpy
    ind = numpy.max(pipe.predict_proba(X_test), axis=1) >= 0.6
    confusion_matrix_df(y_test[ind], pipe.predict(X_test)[ind])






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>3</th>
          <th>4</th>
          <th>5</th>
          <th>6</th>
          <th>7</th>
          <th>8</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0</td>
          <td>0</td>
          <td>2</td>
          <td>1</td>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0</td>
          <td>5</td>
          <td>21</td>
          <td>6</td>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0</td>
          <td>0</td>
          <td>517</td>
          <td>77</td>
          <td>1</td>
          <td>0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0</td>
          <td>0</td>
          <td>109</td>
          <td>539</td>
          <td>7</td>
          <td>0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0</td>
          <td>0</td>
          <td>7</td>
          <td>71</td>
          <td>141</td>
          <td>0</td>
        </tr>
        <tr>
          <th>5</th>
          <td>0</td>
          <td>0</td>
          <td>0</td>
          <td>10</td>
          <td>3</td>
          <td>25</td>
        </tr>
      </tbody>
    </table>
    </div>



Les petites classes ont disparu : le modèle n’est pas sûr du tout pour
les classes 3, 4, 9. On voit aussi que le modèle se trompe souvent d’une
note, il serait sans doute plus judicieux de passer à un modèle de
régression plutôt que de classification. Cependant, un modèle de
régression ne fournit pas de score de confiance. Sans doute serait-il
possible d’en construire avec un modèle de détection d’anomalie…

Anomalies
---------

Une anomalie est un point aberrant. Cela revient à dire que sa
probabilité qu’un tel événement se reproduise est faible. Un modèle
assez connu est
`EllipticEnvelope <https://scikit-learn.org/stable/auto_examples/plot_anomaly_comparison.html>`__.
On suppose que si le modèle détecte une anomalie, un modèle de
prédiction aura plus de mal à prédire. On réutilise le pipeline
précédent en changeant juste la dernière étape.

.. code:: ipython3

    from sklearn.covariance import EllipticEnvelope
    
    one = Pipeline([
        ("prep", ColumnTransformer([
            ("color", Pipeline([
                ('one', OneHotEncoder()),
                ('select', ColumnTransformer([('sel1', 'passthrough', [0])]))
            ]), ['color']),
            ("others", "passthrough", numeric_features)
        ])),
        ("lr", EllipticEnvelope()),
    ])
    
    one.fit(X_train)
    ano = one.predict(X_test)
    ano




.. parsed-literal::
    array([1, 1, 1, ..., 1, 1, 1])



.. code:: ipython3

    from pandas import DataFrame
    df = DataFrame(dict(note=y_test, ano=one.decision_function(X_test), 
                        pred=pipe.predict(X_test), 
                        errors=y_test == pipe.predict(X_test),
                        proba_max=numpy.max(pipe.predict_proba(X_test), axis=1),
                    ))
    df["anoclip"] = df.ano.apply(lambda x: max(x, -200))
    df.head()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>note</th>
          <th>ano</th>
          <th>pred</th>
          <th>errors</th>
          <th>proba_max</th>
          <th>anoclip</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>3776</th>
          <td>7</td>
          <td>114.263113</td>
          <td>5</td>
          <td>False</td>
          <td>0.68</td>
          <td>114.263113</td>
        </tr>
        <tr>
          <th>5800</th>
          <td>6</td>
          <td>113.160338</td>
          <td>6</td>
          <td>True</td>
          <td>0.82</td>
          <td>113.160338</td>
        </tr>
        <tr>
          <th>4028</th>
          <td>6</td>
          <td>107.483432</td>
          <td>6</td>
          <td>True</td>
          <td>0.55</td>
          <td>107.483432</td>
        </tr>
        <tr>
          <th>3311</th>
          <td>6</td>
          <td>112.045947</td>
          <td>5</td>
          <td>False</td>
          <td>0.46</td>
          <td>112.045947</td>
        </tr>
        <tr>
          <th>4716</th>
          <td>7</td>
          <td>118.097584</td>
          <td>6</td>
          <td>False</td>
          <td>0.59</td>
          <td>118.097584</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    import seaborn
    seaborn.lmplot(x="anoclip", y="proba_max", hue="errors",
                   truncate=True, height=5, data=df,
                   logx=True, fit_reg=False);



.. image:: td2a_correction_cl_reg_anomaly_42_0.png


.. code:: ipython3

    df.corr()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>note</th>
          <th>ano</th>
          <th>pred</th>
          <th>errors</th>
          <th>proba_max</th>
          <th>anoclip</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>note</th>
          <td>1.000000</td>
          <td>0.111448</td>
          <td>0.615705</td>
          <td>-0.069233</td>
          <td>-0.080223</td>
          <td>0.167420</td>
        </tr>
        <tr>
          <th>ano</th>
          <td>0.111448</td>
          <td>1.000000</td>
          <td>0.099407</td>
          <td>0.025075</td>
          <td>0.012574</td>
          <td>0.640359</td>
        </tr>
        <tr>
          <th>pred</th>
          <td>0.615705</td>
          <td>0.099407</td>
          <td>1.000000</td>
          <td>-0.035231</td>
          <td>-0.149703</td>
          <td>0.168249</td>
        </tr>
        <tr>
          <th>errors</th>
          <td>-0.069233</td>
          <td>0.025075</td>
          <td>-0.035231</td>
          <td>1.000000</td>
          <td>0.345579</td>
          <td>0.034597</td>
        </tr>
        <tr>
          <th>proba_max</th>
          <td>-0.080223</td>
          <td>0.012574</td>
          <td>-0.149703</td>
          <td>0.345579</td>
          <td>1.000000</td>
          <td>0.008371</td>
        </tr>
        <tr>
          <th>anoclip</th>
          <td>0.167420</td>
          <td>0.640359</td>
          <td>0.168249</td>
          <td>0.034597</td>
          <td>0.008371</td>
          <td>1.000000</td>
        </tr>
      </tbody>
    </table>
    </div>



Les résultats précédents ne sont pas probants. On peut changer de modèle
de détection d’anomalies mais les conclusions restent les mêmes. Le
score d’anomalie n’est pas relié au score de prédiction.

.. code:: ipython3

    fig, ax = plt.subplots(1, 2, figsize=(14, 4))
    df.proba_max.hist(ax=ax[0], bins=50)
    df.anoclip.hist(ax=ax[1], bins=50)
    ax[0].set_title("Distribution du score de classification")
    ax[1].set_title("Distribution du score d'anomalie");



.. image:: td2a_correction_cl_reg_anomaly_45_0.png


C’est joli mais ils n’ont rien à voir. Et c’était prévisible car le
modèle de prédiction qu’on utilise est tout-à-fait capable de prédire ce
qu’est une anomalie.

.. code:: ipython3

    
    pipe_ano = Pipeline([
        ("prep", ColumnTransformer([
            ("color", Pipeline([
                ('one', OneHotEncoder()),
                ('select', ColumnTransformer([('sel1', 'passthrough', [0])]))
            ]), ['color']),
            ("others", "passthrough", numeric_features)
        ])),
        ("lr", RandomForestClassifier()),
    ])
    
    pipe_ano.fit(X_train, one.predict(X_train))
    confusion_matrix_df(one.predict(X_test), pipe_ano.predict(X_test))






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>-1</th>
          <th>1</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>276</td>
          <td>57</td>
        </tr>
        <tr>
          <th>1</th>
          <td>31</td>
          <td>2885</td>
        </tr>
      </tbody>
    </table>
    </div>



Le modèle d’anomalie n’apporte donc aucune information nouvelle. Cela
signifie que le modèle prédictif initial n’améliorerait pas sa
prédiction en utilisant le score d’anomalie. Il n’y a donc aucune chance
que les erreurs ou les score de prédiction soient corrélés au score
d’anomalie d’une manière ou d’une autre.

Score de confiance pour une régression
--------------------------------------

.. code:: ipython3

    from sklearn.ensemble import RandomForestRegressor
    from sklearn.metrics import r2_score
    
    pipe_reg = Pipeline([
        ("prep", ColumnTransformer([
            ("color", Pipeline([
                ('one', OneHotEncoder()),
                ('select', ColumnTransformer([('sel1', 'passthrough', [0])]))
            ]), ['color']),
            ("others", "passthrough", numeric_features)
        ])),
        ("lr", RandomForestRegressor()),
    ])
    
    pipe_reg.fit(X_train, y_train)
    r2_score(y_test, pipe_reg.predict(X_test))




.. parsed-literal::
    0.45446782471634095



Pas super. Mais…

.. code:: ipython3

    error = y_test - pipe_reg.predict(X_test)
    score = numpy.max(pipe.predict_proba(X_test), axis=1)

.. code:: ipython3

    fig, ax = plt.subplots(1, 2, figsize=(12, 4))
    seaborn.kdeplot(score, error, ax=ax[1])
    ax[1].set_ylim([-1.5, 1.5])
    ax[1].set_title("Densité")
    ax[0].plot(score, error, ".")
    ax[0].set_xlabel("score de confiance du classifieur")
    ax[0].set_ylabel("Erreur de prédiction")
    ax[0].set_title("Lien entre classification et prédiction");


.. parsed-literal::
    C:\Python395_x64\lib\site-packages\seaborn\_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
      warnings.warn(



.. image:: td2a_correction_cl_reg_anomaly_53_1.png


Comme prévu le modèle ne se trompe pas plus dans un sens que dans
l’autre.

.. code:: ipython3

    fig, ax = plt.subplots(1, 2, figsize=(12, 4))
    seaborn.kdeplot(score, error.abs(), ax=ax[1])
    ax[1].set_ylim([0, 1.5])
    ax[1].set_title("Densité")
    ax[0].plot(score, error.abs(), ".")
    ax[0].set_xlabel("score de confiance du classifieur")
    ax[0].set_ylabel("Erreur de prédiction")
    ax[0].set_title("Lien entre classification et prédiction");


.. parsed-literal::
    C:\Python395_x64\lib\site-packages\seaborn\_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
      warnings.warn(



.. image:: td2a_correction_cl_reg_anomaly_55_1.png


On vient de construire un indicateur de confiance pour un modèle de
régression. On voit aussi que l’erreur de prédiction est concentrée
autour de 0.5 lorsque le score est faible. C’est normal dans la mesure
où la probabilité est faible lorsque le classifieur n’est pas sûr,
c’est-à-dire que l’observation à prédire est proche de la frontière
entre deux classes. Ces classes sont centrées autour des notes entières,
les frontières sont au milieu, soit approximativement 3.5, 4.5, … Ce
n’est pas une preuve mais seulement la vérifie que l’intervalle de
confiance qu’on vient de fabriquer n’est pas complètement aberrant.