.. _benchmarkrst:

=========
Benchmark
=========


.. only:: html

    **Links:** :download:`notebook <benchmark.ipynb>`, :downloadlink:`html <benchmark2html.html>`, :download:`PDF <benchmark.pdf>`, :download:`python <benchmark.py>`, :downloadlink:`slides <benchmark.slides.html>`, :githublink:`GitHub|_doc/notebooks/ml/benchmark.ipynb|*`


Ce notebook compare différents modèles depuis un notebook.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





Si le message *Widget Javascript not detected. It may not be installed
or enabled properly.* apparaît, vous devriez exécuter la commande
``jupyter nbextension enable --py --sys-prefix widgetsnbextension``
depuis la ligne de commande. Le code suivant vous permet de vérifier que
cela a été fait.

.. code:: ipython3

    from tqdm import tnrange, tqdm_notebook
    from time import sleep
    
    for i in tnrange(3, desc='1st loop'):
        for j in tqdm_notebook(range(20), desc='2nd loop'):
            sleep(0.01)










.. parsed-literal::

    


.. code:: ipython3

    %matplotlib inline

Petit bench sur le clustering
-----------------------------

Définition du bench
~~~~~~~~~~~~~~~~~~~

.. code:: ipython3

    import dill
    from tqdm import tnrange
    from sklearn.cluster import AgglomerativeClustering, KMeans
    from sklearn.datasets import make_blobs
    from mlstatpy.ml import MlGridBenchMark
    
    params = [dict(model=lambda : KMeans(n_clusters=3), name="KMeans-3", shortname="km-3"),
              dict(model=lambda : AgglomerativeClustering(), name="AgglomerativeClustering", shortname="aggclus")]
    
    datasets = [dict(X=make_blobs(100, centers=3)[0], Nclus=3,
                     name="blob-100-3", shortname="b-100-3", no_split=True),
                dict(X=make_blobs(100, centers=5)[0], Nclus=5, 
                     name="blob-100-5", shortname="b-100-5", no_split=True) ]
    
    bench = MlGridBenchMark("TestName", datasets, fLOG=None, clog=None,
                            cache_file="cache.pickle", pickle_module=dill, 
                            repetition=3, progressbar=tnrange,
                            graphx=["_time", "time_train", "Nclus"],
                            graphy=["silhouette", "Nrows"])

Lancer le bench
~~~~~~~~~~~~~~~

.. code:: ipython3

    bench.run(params)




.. parsed-literal::
    0/|/2017-03-19 20:11:11 [BenchMark.run] number of cached run: 4:   0%|| 0/4 [00:00<?, ?it/s]                           



.. parsed-literal::

    3/|/2017-03-19 20:11:13 [BenchMark.run] done.:  75%|| 3/4 [00:02<00:00,  1.10it/s]                                     11it/s]_train': 0.02142968022685221, 'time_test': 0.0025012412126208527, '_btry': 'aggclus-b-100-5', '_iexp': 2, 'model_name': 'AgglomerativeClustering', 'ds_name': 'blob-100-5', 'Nrows': 100, 'Nfeat': 2, 'Nclus': 5, 'no_split': True, '_date': datetime.datetime(2017, 3, 19, 20, 11, 11, 647355), '_time': 0.1007650830318858, '_span': datetime.timedelta(0, 0, 112581), '_i': 3, '_name': 'TestName'}:  75%|| 3/4 [00:00<00:00,  4.22it/s]]0:00,  3.53it/s]]


Récupérer les résultats
~~~~~~~~~~~~~~~~~~~~~~~

.. code:: ipython3

    df = bench.to_df()
    df






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>_btry</th>
          <th>_date</th>
          <th>_i</th>
          <th>_iexp</th>
          <th>_name</th>
          <th>_span</th>
          <th>_time</th>
          <th>Nclus</th>
          <th>Nfeat</th>
          <th>Nrows</th>
          <th>ds_name</th>
          <th>model_name</th>
          <th>no_split</th>
          <th>own_score</th>
          <th>silhouette</th>
          <th>time_preproc</th>
          <th>time_test</th>
          <th>time_train</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>km-3-b-100-3</td>
          <td>2017-03-19 20:11:11.132135</td>
          <td>0</td>
          <td>0</td>
          <td>TestName</td>
          <td>0:00:00.147610</td>
          <td>0.147594</td>
          <td>3</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-3</td>
          <td>KMeans-3</td>
          <td>True</td>
          <td>-175.396944</td>
          <td>0.700618</td>
          <td>0.009154</td>
          <td>0.003195</td>
          <td>0.044693</td>
        </tr>
        <tr>
          <th>1</th>
          <td>km-3-b-100-3</td>
          <td>0:00:00.147610</td>
          <td>0</td>
          <td>1</td>
          <td>TestName</td>
          <td>2017-03-19 20:11:11.140141</td>
          <td>0.147594</td>
          <td>3</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-3</td>
          <td>KMeans-3</td>
          <td>True</td>
          <td>-175.396944</td>
          <td>0.700618</td>
          <td>0.006068</td>
          <td>0.002803</td>
          <td>0.037633</td>
        </tr>
        <tr>
          <th>2</th>
          <td>km-3-b-100-3</td>
          <td>2017-03-19 20:11:11.140141</td>
          <td>0</td>
          <td>2</td>
          <td>TestName</td>
          <td>0:00:00.155620</td>
          <td>0.147594</td>
          <td>3</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-3</td>
          <td>KMeans-3</td>
          <td>True</td>
          <td>-175.396944</td>
          <td>0.700618</td>
          <td>0.006230</td>
          <td>0.002630</td>
          <td>0.035106</td>
        </tr>
        <tr>
          <th>3</th>
          <td>aggclus-b-100-3</td>
          <td>2017-03-19 20:11:11.317283</td>
          <td>1</td>
          <td>0</td>
          <td>TestName</td>
          <td>0:00:00.096081</td>
          <td>0.096700</td>
          <td>3</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-3</td>
          <td>AgglomerativeClustering</td>
          <td>True</td>
          <td>NaN</td>
          <td>0.662345</td>
          <td>0.008147</td>
          <td>0.002508</td>
          <td>0.026997</td>
        </tr>
        <tr>
          <th>4</th>
          <td>aggclus-b-100-3</td>
          <td>0:00:00.096081</td>
          <td>1</td>
          <td>1</td>
          <td>TestName</td>
          <td>2017-03-19 20:11:11.325288</td>
          <td>0.096700</td>
          <td>3</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-3</td>
          <td>AgglomerativeClustering</td>
          <td>True</td>
          <td>NaN</td>
          <td>0.662345</td>
          <td>0.009511</td>
          <td>0.004156</td>
          <td>0.016807</td>
        </tr>
        <tr>
          <th>5</th>
          <td>aggclus-b-100-3</td>
          <td>2017-03-19 20:11:11.325288</td>
          <td>1</td>
          <td>2</td>
          <td>TestName</td>
          <td>0:00:00.106088</td>
          <td>0.096700</td>
          <td>3</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-3</td>
          <td>AgglomerativeClustering</td>
          <td>True</td>
          <td>NaN</td>
          <td>0.662345</td>
          <td>0.007018</td>
          <td>0.003252</td>
          <td>0.018227</td>
        </tr>
        <tr>
          <th>6</th>
          <td>km-3-b-100-5</td>
          <td>2017-03-19 20:11:11.452688</td>
          <td>2</td>
          <td>0</td>
          <td>TestName</td>
          <td>0:00:00.145130</td>
          <td>0.145012</td>
          <td>5</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-5</td>
          <td>KMeans-3</td>
          <td>True</td>
          <td>-466.829200</td>
          <td>0.790511</td>
          <td>0.007587</td>
          <td>0.002748</td>
          <td>0.033610</td>
        </tr>
        <tr>
          <th>7</th>
          <td>km-3-b-100-5</td>
          <td>0:00:00.145130</td>
          <td>2</td>
          <td>1</td>
          <td>TestName</td>
          <td>2017-03-19 20:11:11.463199</td>
          <td>0.145012</td>
          <td>5</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-5</td>
          <td>KMeans-3</td>
          <td>True</td>
          <td>-466.829200</td>
          <td>0.790511</td>
          <td>0.007471</td>
          <td>0.002278</td>
          <td>0.036098</td>
        </tr>
        <tr>
          <th>8</th>
          <td>km-3-b-100-5</td>
          <td>2017-03-19 20:11:11.463199</td>
          <td>2</td>
          <td>2</td>
          <td>TestName</td>
          <td>0:00:00.153136</td>
          <td>0.145012</td>
          <td>5</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-5</td>
          <td>KMeans-3</td>
          <td>True</td>
          <td>-466.829200</td>
          <td>0.790511</td>
          <td>0.011576</td>
          <td>0.004463</td>
          <td>0.039103</td>
        </tr>
        <tr>
          <th>9</th>
          <td>aggclus-b-100-5</td>
          <td>2017-03-19 20:11:11.640351</td>
          <td>3</td>
          <td>0</td>
          <td>TestName</td>
          <td>0:00:00.101573</td>
          <td>0.100765</td>
          <td>5</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-5</td>
          <td>AgglomerativeClustering</td>
          <td>True</td>
          <td>NaN</td>
          <td>0.636241</td>
          <td>0.009483</td>
          <td>0.002418</td>
          <td>0.020562</td>
        </tr>
        <tr>
          <th>10</th>
          <td>aggclus-b-100-5</td>
          <td>0:00:00.101573</td>
          <td>3</td>
          <td>1</td>
          <td>TestName</td>
          <td>2017-03-19 20:11:11.647355</td>
          <td>0.100765</td>
          <td>5</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-5</td>
          <td>AgglomerativeClustering</td>
          <td>True</td>
          <td>NaN</td>
          <td>0.636241</td>
          <td>0.011532</td>
          <td>0.001634</td>
          <td>0.021456</td>
        </tr>
        <tr>
          <th>11</th>
          <td>aggclus-b-100-5</td>
          <td>2017-03-19 20:11:11.647355</td>
          <td>3</td>
          <td>2</td>
          <td>TestName</td>
          <td>0:00:00.112581</td>
          <td>0.100765</td>
          <td>5</td>
          <td>2</td>
          <td>100</td>
          <td>blob-100-5</td>
          <td>AgglomerativeClustering</td>
          <td>True</td>
          <td>NaN</td>
          <td>0.636241</td>
          <td>0.009643</td>
          <td>0.002501</td>
          <td>0.021430</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    df.plot(x="time_train", y="silhouette", kind="scatter")




.. parsed-literal::
    <matplotlib.axes._subplots.AxesSubplot at 0x122b8004748>




.. image:: benchmark_12_1.png


Dessin, Graphs
~~~~~~~~~~~~~~

.. code:: ipython3

    bench.plot_graphs(figsize=(12,12))




.. parsed-literal::
    array([[<matplotlib.axes._subplots.AxesSubplot object at 0x00000122B8269A90>,
            <matplotlib.axes._subplots.AxesSubplot object at 0x00000122B82E1DA0>],
           [<matplotlib.axes._subplots.AxesSubplot object at 0x00000122B83512E8>,
            <matplotlib.axes._subplots.AxesSubplot object at 0x00000122B83A1828>],
           [<matplotlib.axes._subplots.AxesSubplot object at 0x00000122B8409D68>,
            <matplotlib.axes._subplots.AxesSubplot object at 0x00000122B8462588>]], dtype=object)




.. image:: benchmark_14_1.png