.. _onlinenewspopylarityrst:

===================================
OnlineNewPopularity (data from UCI)
===================================


.. only:: html

    **Links:** :download:`notebook <online_news_popylarity.ipynb>`, :downloadlink:`html <online_news_popylarity2html.html>`, :download:`PDF <online_news_popylarity.pdf>`, :download:`python <online_news_popylarity.py>`, :downloadlink:`slides <online_news_popylarity.slides.html>`, :githublink:`GitHub|_doc/notebooks/mlexamples/online_news_popylarity.ipynb|*`


This notebook suggests a couple of ways to explore the data of a machine
learning problem.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





.. code:: ipython3

    %matplotlib inline
    import matplotlib.pyplot as plt
    import seaborn
    import numpy
    import pandas

Download data
-------------

`Online News Popularity Data
Set <https://archive.ics.uci.edu/ml/datasets/Online+News+Popularity>`__

.. code:: ipython3

    import pyensae.datasource
    pyensae.datasource.download_data("OnlineNewsPopularity.zip",
                          url="https://archive.ics.uci.edu/ml/machine-learning-databases/00332/")




.. parsed-literal::
    ['OnlineNewsPopularity/OnlineNewsPopularity.names',
     'OnlineNewsPopularity/OnlineNewsPopularity.csv']



.. code:: ipython3

    data = pandas.read_csv("OnlineNewsPopularity/OnlineNewsPopularity.csv")
    data.columns = [c.strip() for c in data.columns]  # remove spaces around data
    data.head()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>url</th>
          <th>timedelta</th>
          <th>n_tokens_title</th>
          <th>n_tokens_content</th>
          <th>n_unique_tokens</th>
          <th>n_non_stop_words</th>
          <th>n_non_stop_unique_tokens</th>
          <th>num_hrefs</th>
          <th>num_self_hrefs</th>
          <th>num_imgs</th>
          <th>...</th>
          <th>min_positive_polarity</th>
          <th>max_positive_polarity</th>
          <th>avg_negative_polarity</th>
          <th>min_negative_polarity</th>
          <th>max_negative_polarity</th>
          <th>title_subjectivity</th>
          <th>title_sentiment_polarity</th>
          <th>abs_title_subjectivity</th>
          <th>abs_title_sentiment_polarity</th>
          <th>shares</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>http://mashable.com/2013/01/07/amazon-instant-...</td>
          <td>731.0</td>
          <td>12.0</td>
          <td>219.0</td>
          <td>0.663594</td>
          <td>1.0</td>
          <td>0.815385</td>
          <td>4.0</td>
          <td>2.0</td>
          <td>1.0</td>
          <td>...</td>
          <td>0.100000</td>
          <td>0.7</td>
          <td>-0.350000</td>
          <td>-0.600</td>
          <td>-0.200000</td>
          <td>0.500000</td>
          <td>-0.187500</td>
          <td>0.000000</td>
          <td>0.187500</td>
          <td>593</td>
        </tr>
        <tr>
          <th>1</th>
          <td>http://mashable.com/2013/01/07/ap-samsung-spon...</td>
          <td>731.0</td>
          <td>9.0</td>
          <td>255.0</td>
          <td>0.604743</td>
          <td>1.0</td>
          <td>0.791946</td>
          <td>3.0</td>
          <td>1.0</td>
          <td>1.0</td>
          <td>...</td>
          <td>0.033333</td>
          <td>0.7</td>
          <td>-0.118750</td>
          <td>-0.125</td>
          <td>-0.100000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.500000</td>
          <td>0.000000</td>
          <td>711</td>
        </tr>
        <tr>
          <th>2</th>
          <td>http://mashable.com/2013/01/07/apple-40-billio...</td>
          <td>731.0</td>
          <td>9.0</td>
          <td>211.0</td>
          <td>0.575130</td>
          <td>1.0</td>
          <td>0.663866</td>
          <td>3.0</td>
          <td>1.0</td>
          <td>1.0</td>
          <td>...</td>
          <td>0.100000</td>
          <td>1.0</td>
          <td>-0.466667</td>
          <td>-0.800</td>
          <td>-0.133333</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.500000</td>
          <td>0.000000</td>
          <td>1500</td>
        </tr>
        <tr>
          <th>3</th>
          <td>http://mashable.com/2013/01/07/astronaut-notre...</td>
          <td>731.0</td>
          <td>9.0</td>
          <td>531.0</td>
          <td>0.503788</td>
          <td>1.0</td>
          <td>0.665635</td>
          <td>9.0</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>...</td>
          <td>0.136364</td>
          <td>0.8</td>
          <td>-0.369697</td>
          <td>-0.600</td>
          <td>-0.166667</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.500000</td>
          <td>0.000000</td>
          <td>1200</td>
        </tr>
        <tr>
          <th>4</th>
          <td>http://mashable.com/2013/01/07/att-u-verse-apps/</td>
          <td>731.0</td>
          <td>13.0</td>
          <td>1072.0</td>
          <td>0.415646</td>
          <td>1.0</td>
          <td>0.540890</td>
          <td>19.0</td>
          <td>19.0</td>
          <td>20.0</td>
          <td>...</td>
          <td>0.033333</td>
          <td>1.0</td>
          <td>-0.220192</td>
          <td>-0.500</td>
          <td>-0.050000</td>
          <td>0.454545</td>
          <td>0.136364</td>
          <td>0.045455</td>
          <td>0.136364</td>
          <td>505</td>
        </tr>
      </tbody>
    </table>
    <p>5 rows × 61 columns</p>
    </div>



.. code:: ipython3

    data.shape




.. parsed-literal::
    (39644, 61)



.. code:: ipython3

    import numpy
    numeric = [c for i,c in enumerate(data.columns) if data.dtypes[i] in [numpy.float64, numpy.int64]]
    len(numeric)




.. parsed-literal::
    60



Corr-Pair-Plots and scales
--------------------------

.. code:: ipython3

    cmap = seaborn.diverging_palette(h_neg=210, h_pos=350, s=90, l=30, as_cmap=True, center="light")
    seaborn.clustermap(data[numeric].corr(), figsize=(14, 14), cmap=cmap);



.. image:: online_news_popylarity_9_0.png


.. code:: ipython3

    numeric[:5]




.. parsed-literal::
    ['timedelta',
     'n_tokens_title',
     'n_tokens_content',
     'n_unique_tokens',
     'n_non_stop_words']



.. code:: ipython3

    data_numeric5 = data[numeric[:5]]

.. code:: ipython3

    seaborn.clustermap(data_numeric5.corr(), figsize=(6, 6), cmap=cmap);



.. image:: online_news_popylarity_12_0.png


.. code:: ipython3

    data_numeric5[::100].shape




.. parsed-literal::
    (397, 5)



We take a subsample as the whole dataframe takes time to plot.

.. code:: ipython3

    def my_pair_plot(df):
        g = seaborn.PairGrid(df, diag_sharey=False)
        g.map_upper(seaborn.scatterplot, s=15)
        g.map_lower(seaborn.kdeplot)
        g.map_diag(seaborn.kdeplot, lw=2)
        return g
    
    my_pair_plot(data_numeric5[::100]);



.. image:: online_news_popylarity_15_0.png


Or maybe it is because there are some outliers.

.. code:: ipython3

    data[data.n_unique_tokens > 100]






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>url</th>
          <th>timedelta</th>
          <th>n_tokens_title</th>
          <th>n_tokens_content</th>
          <th>n_unique_tokens</th>
          <th>n_non_stop_words</th>
          <th>n_non_stop_unique_tokens</th>
          <th>num_hrefs</th>
          <th>num_self_hrefs</th>
          <th>num_imgs</th>
          <th>...</th>
          <th>min_positive_polarity</th>
          <th>max_positive_polarity</th>
          <th>avg_negative_polarity</th>
          <th>min_negative_polarity</th>
          <th>max_negative_polarity</th>
          <th>title_subjectivity</th>
          <th>title_sentiment_polarity</th>
          <th>abs_title_subjectivity</th>
          <th>abs_title_sentiment_polarity</th>
          <th>shares</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>31037</th>
          <td>http://mashable.com/2014/08/18/ukraine-civilia...</td>
          <td>142.0</td>
          <td>9.0</td>
          <td>1570.0</td>
          <td>701.0</td>
          <td>1042.0</td>
          <td>650.0</td>
          <td>11.0</td>
          <td>10.0</td>
          <td>51.0</td>
          <td>...</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>5900</td>
        </tr>
      </tbody>
    </table>
    <p>1 rows × 61 columns</p>
    </div>



We remove this row as it seems an outliar:

.. code:: ipython3

    data_clean = data[data.n_unique_tokens < 100].copy()

.. code:: ipython3

    my_pair_plot(data_clean[numeric[:5]][::100]);



.. image:: online_news_popylarity_20_0.png


.. code:: ipython3

    data_clean.hist(figsize=(16,16));



.. image:: online_news_popylarity_21_0.png


.. code:: ipython3

    desc = data_clean.describe().T
    desc["log"] = (desc["max"] > desc["50%"] * 9) & (desc["max"] > 1)
    desc["scale"] = ""
    desc.loc[desc["log"],"scale"] = "log"
    desc[["mean", "min", "50%", "max", "scale"]]






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>mean</th>
          <th>min</th>
          <th>50%</th>
          <th>max</th>
          <th>scale</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>timedelta</th>
          <td>354.535832</td>
          <td>8.000000</td>
          <td>339.000000</td>
          <td>731.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>n_tokens_title</th>
          <td>10.398784</td>
          <td>2.000000</td>
          <td>10.000000</td>
          <td>23.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>n_tokens_content</th>
          <td>546.488914</td>
          <td>0.000000</td>
          <td>409.000000</td>
          <td>8474.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>n_unique_tokens</th>
          <td>0.530547</td>
          <td>0.000000</td>
          <td>0.539216</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>n_non_stop_words</th>
          <td>0.970209</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>n_non_stop_unique_tokens</th>
          <td>0.672796</td>
          <td>0.000000</td>
          <td>0.690476</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>num_hrefs</th>
          <td>10.883687</td>
          <td>0.000000</td>
          <td>8.000000</td>
          <td>304.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>num_self_hrefs</th>
          <td>3.293469</td>
          <td>0.000000</td>
          <td>3.000000</td>
          <td>116.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>num_imgs</th>
          <td>4.542971</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>128.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>num_videos</th>
          <td>1.249905</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>91.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>average_token_length</th>
          <td>4.548236</td>
          <td>0.000000</td>
          <td>4.664078</td>
          <td>8.041534</td>
          <td></td>
        </tr>
        <tr>
          <th>num_keywords</th>
          <td>7.223772</td>
          <td>1.000000</td>
          <td>7.000000</td>
          <td>10.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_lifestyle</th>
          <td>0.052948</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_entertainment</th>
          <td>0.177989</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_bus</th>
          <td>0.157859</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_socmed</th>
          <td>0.058598</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_tech</th>
          <td>0.185304</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_world</th>
          <td>0.212572</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>kw_min_min</th>
          <td>26.107484</td>
          <td>-1.000000</td>
          <td>-1.000000</td>
          <td>377.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>kw_max_min</th>
          <td>1153.961166</td>
          <td>0.000000</td>
          <td>660.000000</td>
          <td>298400.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>kw_avg_min</th>
          <td>312.371221</td>
          <td>-1.000000</td>
          <td>235.500000</td>
          <td>42827.857143</td>
          <td>log</td>
        </tr>
        <tr>
          <th>kw_min_max</th>
          <td>13612.114774</td>
          <td>0.000000</td>
          <td>1400.000000</td>
          <td>843300.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>kw_max_max</th>
          <td>752321.771813</td>
          <td>0.000000</td>
          <td>843300.000000</td>
          <td>843300.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>kw_avg_max</th>
          <td>259280.143039</td>
          <td>0.000000</td>
          <td>244566.666667</td>
          <td>843300.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>kw_min_avg</th>
          <td>1117.113731</td>
          <td>-1.000000</td>
          <td>1023.619048</td>
          <td>3613.039820</td>
          <td></td>
        </tr>
        <tr>
          <th>kw_max_avg</th>
          <td>5657.265804</td>
          <td>0.000000</td>
          <td>4355.694105</td>
          <td>298400.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>kw_avg_avg</th>
          <td>3135.864283</td>
          <td>0.000000</td>
          <td>2870.047184</td>
          <td>43567.659946</td>
          <td>log</td>
        </tr>
        <tr>
          <th>self_reference_min_shares</th>
          <td>3998.836211</td>
          <td>0.000000</td>
          <td>1200.000000</td>
          <td>843300.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>self_reference_max_shares</th>
          <td>10329.473218</td>
          <td>0.000000</td>
          <td>2800.000000</td>
          <td>843300.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>self_reference_avg_sharess</th>
          <td>6401.684395</td>
          <td>0.000000</td>
          <td>2200.000000</td>
          <td>843300.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>weekday_is_monday</th>
          <td>0.168025</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_tuesday</th>
          <td>0.186389</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_wednesday</th>
          <td>0.187549</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_thursday</th>
          <td>0.183311</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_friday</th>
          <td>0.143808</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_saturday</th>
          <td>0.061877</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_sunday</th>
          <td>0.069041</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>is_weekend</th>
          <td>0.130918</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>LDA_00</th>
          <td>0.184604</td>
          <td>0.018182</td>
          <td>0.033387</td>
          <td>0.926994</td>
          <td></td>
        </tr>
        <tr>
          <th>LDA_01</th>
          <td>0.141259</td>
          <td>0.018182</td>
          <td>0.033345</td>
          <td>0.925947</td>
          <td></td>
        </tr>
        <tr>
          <th>LDA_02</th>
          <td>0.216326</td>
          <td>0.018182</td>
          <td>0.040004</td>
          <td>0.919999</td>
          <td></td>
        </tr>
        <tr>
          <th>LDA_03</th>
          <td>0.223775</td>
          <td>0.018182</td>
          <td>0.040001</td>
          <td>0.926534</td>
          <td></td>
        </tr>
        <tr>
          <th>LDA_04</th>
          <td>0.234035</td>
          <td>0.018182</td>
          <td>0.040727</td>
          <td>0.927191</td>
          <td></td>
        </tr>
        <tr>
          <th>global_subjectivity</th>
          <td>0.443381</td>
          <td>0.000000</td>
          <td>0.453458</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>global_sentiment_polarity</th>
          <td>0.119312</td>
          <td>-0.393750</td>
          <td>0.119119</td>
          <td>0.727841</td>
          <td></td>
        </tr>
        <tr>
          <th>global_rate_positive_words</th>
          <td>0.039626</td>
          <td>0.000000</td>
          <td>0.039024</td>
          <td>0.155488</td>
          <td></td>
        </tr>
        <tr>
          <th>global_rate_negative_words</th>
          <td>0.016613</td>
          <td>0.000000</td>
          <td>0.015337</td>
          <td>0.184932</td>
          <td></td>
        </tr>
        <tr>
          <th>rate_positive_words</th>
          <td>0.682167</td>
          <td>0.000000</td>
          <td>0.710526</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>rate_negative_words</th>
          <td>0.287941</td>
          <td>0.000000</td>
          <td>0.280000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>avg_positive_polarity</th>
          <td>0.353834</td>
          <td>0.000000</td>
          <td>0.358760</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>min_positive_polarity</th>
          <td>0.095448</td>
          <td>0.000000</td>
          <td>0.100000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>max_positive_polarity</th>
          <td>0.756747</td>
          <td>0.000000</td>
          <td>0.800000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>avg_negative_polarity</th>
          <td>-0.259531</td>
          <td>-1.000000</td>
          <td>-0.253333</td>
          <td>0.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>min_negative_polarity</th>
          <td>-0.521957</td>
          <td>-1.000000</td>
          <td>-0.500000</td>
          <td>0.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>max_negative_polarity</th>
          <td>-0.107503</td>
          <td>-1.000000</td>
          <td>-0.100000</td>
          <td>0.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>title_subjectivity</th>
          <td>0.282360</td>
          <td>0.000000</td>
          <td>0.150000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>title_sentiment_polarity</th>
          <td>0.071427</td>
          <td>-1.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>abs_title_subjectivity</th>
          <td>0.341851</td>
          <td>0.000000</td>
          <td>0.500000</td>
          <td>0.500000</td>
          <td></td>
        </tr>
        <tr>
          <th>abs_title_sentiment_polarity</th>
          <td>0.156068</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>shares</th>
          <td>3395.317004</td>
          <td>1.000000</td>
          <td>1400.000000</td>
          <td>843300.000000</td>
          <td>log</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    numpy.log(data_clean["shares"]).hist(bins=50);



.. image:: online_news_popylarity_23_0.png


.. code:: ipython3

    shares = data_clean[[c for c in numeric if "share" in c]].copy()
    for c in shares.columns:
        shares[c] = numpy.log(shares[c] + 1)
    seaborn.pairplot(shares);



.. image:: online_news_popylarity_24_0.png


.. code:: ipython3

    kw = data_clean[[c for c in numeric if "kw" in c]].copy()
    seaborn.pairplot(kw);



.. image:: online_news_popylarity_25_0.png


Outcome, cleaning scaling
-------------------------

cleaning

.. code:: ipython3

    data_clean = data[data.n_unique_tokens < 100].copy()

scaling: we consider that if the maximum is far away from the mediane,
the scale should be logarithmic as it is far way from a gaussian law, it
just applies on this problem

.. code:: ipython3

    desc = data_clean.describe().T
    desc["log"] = (desc["max"] > desc["50%"] * 9) & (desc["max"] > 1)
    desc["log+2"] = desc["log"] & (desc["min"] < 0)
    desc["scale"] = ""
    desc.loc[desc["log"],"scale"] = "log"
    desc.loc[desc["log+2"],"scale"] = "log+2"
    desc[["mean", "min", "50%", "max", "scale"]]






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>mean</th>
          <th>min</th>
          <th>50%</th>
          <th>max</th>
          <th>scale</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>timedelta</th>
          <td>354.535832</td>
          <td>8.000000</td>
          <td>339.000000</td>
          <td>731.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>n_tokens_title</th>
          <td>10.398784</td>
          <td>2.000000</td>
          <td>10.000000</td>
          <td>23.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>n_tokens_content</th>
          <td>546.488914</td>
          <td>0.000000</td>
          <td>409.000000</td>
          <td>8474.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>n_unique_tokens</th>
          <td>0.530547</td>
          <td>0.000000</td>
          <td>0.539216</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>n_non_stop_words</th>
          <td>0.970209</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>n_non_stop_unique_tokens</th>
          <td>0.672796</td>
          <td>0.000000</td>
          <td>0.690476</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>num_hrefs</th>
          <td>10.883687</td>
          <td>0.000000</td>
          <td>8.000000</td>
          <td>304.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>num_self_hrefs</th>
          <td>3.293469</td>
          <td>0.000000</td>
          <td>3.000000</td>
          <td>116.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>num_imgs</th>
          <td>4.542971</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>128.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>num_videos</th>
          <td>1.249905</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>91.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>average_token_length</th>
          <td>4.548236</td>
          <td>0.000000</td>
          <td>4.664078</td>
          <td>8.041534</td>
          <td></td>
        </tr>
        <tr>
          <th>num_keywords</th>
          <td>7.223772</td>
          <td>1.000000</td>
          <td>7.000000</td>
          <td>10.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_lifestyle</th>
          <td>0.052948</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_entertainment</th>
          <td>0.177989</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_bus</th>
          <td>0.157859</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_socmed</th>
          <td>0.058598</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_tech</th>
          <td>0.185304</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>data_channel_is_world</th>
          <td>0.212572</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>kw_min_min</th>
          <td>26.107484</td>
          <td>-1.000000</td>
          <td>-1.000000</td>
          <td>377.000000</td>
          <td>log+2</td>
        </tr>
        <tr>
          <th>kw_max_min</th>
          <td>1153.961166</td>
          <td>0.000000</td>
          <td>660.000000</td>
          <td>298400.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>kw_avg_min</th>
          <td>312.371221</td>
          <td>-1.000000</td>
          <td>235.500000</td>
          <td>42827.857143</td>
          <td>log+2</td>
        </tr>
        <tr>
          <th>kw_min_max</th>
          <td>13612.114774</td>
          <td>0.000000</td>
          <td>1400.000000</td>
          <td>843300.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>kw_max_max</th>
          <td>752321.771813</td>
          <td>0.000000</td>
          <td>843300.000000</td>
          <td>843300.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>kw_avg_max</th>
          <td>259280.143039</td>
          <td>0.000000</td>
          <td>244566.666667</td>
          <td>843300.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>kw_min_avg</th>
          <td>1117.113731</td>
          <td>-1.000000</td>
          <td>1023.619048</td>
          <td>3613.039820</td>
          <td></td>
        </tr>
        <tr>
          <th>kw_max_avg</th>
          <td>5657.265804</td>
          <td>0.000000</td>
          <td>4355.694105</td>
          <td>298400.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>kw_avg_avg</th>
          <td>3135.864283</td>
          <td>0.000000</td>
          <td>2870.047184</td>
          <td>43567.659946</td>
          <td>log</td>
        </tr>
        <tr>
          <th>self_reference_min_shares</th>
          <td>3998.836211</td>
          <td>0.000000</td>
          <td>1200.000000</td>
          <td>843300.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>self_reference_max_shares</th>
          <td>10329.473218</td>
          <td>0.000000</td>
          <td>2800.000000</td>
          <td>843300.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>self_reference_avg_sharess</th>
          <td>6401.684395</td>
          <td>0.000000</td>
          <td>2200.000000</td>
          <td>843300.000000</td>
          <td>log</td>
        </tr>
        <tr>
          <th>weekday_is_monday</th>
          <td>0.168025</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_tuesday</th>
          <td>0.186389</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_wednesday</th>
          <td>0.187549</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_thursday</th>
          <td>0.183311</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_friday</th>
          <td>0.143808</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_saturday</th>
          <td>0.061877</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>weekday_is_sunday</th>
          <td>0.069041</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>is_weekend</th>
          <td>0.130918</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>LDA_00</th>
          <td>0.184604</td>
          <td>0.018182</td>
          <td>0.033387</td>
          <td>0.926994</td>
          <td></td>
        </tr>
        <tr>
          <th>LDA_01</th>
          <td>0.141259</td>
          <td>0.018182</td>
          <td>0.033345</td>
          <td>0.925947</td>
          <td></td>
        </tr>
        <tr>
          <th>LDA_02</th>
          <td>0.216326</td>
          <td>0.018182</td>
          <td>0.040004</td>
          <td>0.919999</td>
          <td></td>
        </tr>
        <tr>
          <th>LDA_03</th>
          <td>0.223775</td>
          <td>0.018182</td>
          <td>0.040001</td>
          <td>0.926534</td>
          <td></td>
        </tr>
        <tr>
          <th>LDA_04</th>
          <td>0.234035</td>
          <td>0.018182</td>
          <td>0.040727</td>
          <td>0.927191</td>
          <td></td>
        </tr>
        <tr>
          <th>global_subjectivity</th>
          <td>0.443381</td>
          <td>0.000000</td>
          <td>0.453458</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>global_sentiment_polarity</th>
          <td>0.119312</td>
          <td>-0.393750</td>
          <td>0.119119</td>
          <td>0.727841</td>
          <td></td>
        </tr>
        <tr>
          <th>global_rate_positive_words</th>
          <td>0.039626</td>
          <td>0.000000</td>
          <td>0.039024</td>
          <td>0.155488</td>
          <td></td>
        </tr>
        <tr>
          <th>global_rate_negative_words</th>
          <td>0.016613</td>
          <td>0.000000</td>
          <td>0.015337</td>
          <td>0.184932</td>
          <td></td>
        </tr>
        <tr>
          <th>rate_positive_words</th>
          <td>0.682167</td>
          <td>0.000000</td>
          <td>0.710526</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>rate_negative_words</th>
          <td>0.287941</td>
          <td>0.000000</td>
          <td>0.280000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>avg_positive_polarity</th>
          <td>0.353834</td>
          <td>0.000000</td>
          <td>0.358760</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>min_positive_polarity</th>
          <td>0.095448</td>
          <td>0.000000</td>
          <td>0.100000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>max_positive_polarity</th>
          <td>0.756747</td>
          <td>0.000000</td>
          <td>0.800000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>avg_negative_polarity</th>
          <td>-0.259531</td>
          <td>-1.000000</td>
          <td>-0.253333</td>
          <td>0.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>min_negative_polarity</th>
          <td>-0.521957</td>
          <td>-1.000000</td>
          <td>-0.500000</td>
          <td>0.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>max_negative_polarity</th>
          <td>-0.107503</td>
          <td>-1.000000</td>
          <td>-0.100000</td>
          <td>0.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>title_subjectivity</th>
          <td>0.282360</td>
          <td>0.000000</td>
          <td>0.150000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>title_sentiment_polarity</th>
          <td>0.071427</td>
          <td>-1.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>abs_title_subjectivity</th>
          <td>0.341851</td>
          <td>0.000000</td>
          <td>0.500000</td>
          <td>0.500000</td>
          <td></td>
        </tr>
        <tr>
          <th>abs_title_sentiment_polarity</th>
          <td>0.156068</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td></td>
        </tr>
        <tr>
          <th>shares</th>
          <td>3395.317004</td>
          <td>1.000000</td>
          <td>1400.000000</td>
          <td>843300.000000</td>
          <td>log</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    import numpy
    new_data = data_clean.copy()
    for c in desc.index [ desc["scale"] == "log"]:
        new_data[c] = numpy.log(new_data[c] + 1)
    for c in desc.index [ desc["scale"] == "log+2"]:
        new_data[c] = numpy.log(new_data[c] + 2)    

.. code:: ipython3

    new_data.shape




.. parsed-literal::
    (39643, 61)



.. code:: ipython3

    set(new_data.dtypes)




.. parsed-literal::
    {dtype('float64'), dtype('O')}



.. code:: ipython3

    from sklearn.model_selection import train_test_split
    
    features = new_data[[c for c in numeric if c != "shares"]]
    target = new_data["shares"]
    X_train, X_test, y_train, y_test = train_test_split(features, target)

learning
--------

`RandomForestRegressor <http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html>`__

.. code:: ipython3

    from sklearn.ensemble import RandomForestRegressor
    clr = RandomForestRegressor(min_samples_leaf=20, n_estimators=50, min_weight_fraction_leaf=0.01, min_samples_split=10)
    clr.fit(X_train, y_train)




.. parsed-literal::
    RandomForestRegressor(min_samples_leaf=20, min_samples_split=10,
                          min_weight_fraction_leaf=0.01, n_estimators=50)



.. code:: ipython3

    tpredicted = clr.predict(X_train)
    df = pandas.DataFrame()
    df["train_predicted"] = tpredicted
    df["train_expected"] = y_train
    df.corr()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>train_predicted</th>
          <th>train_expected</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>train_predicted</th>
          <td>1.000000</td>
          <td>0.004091</td>
        </tr>
        <tr>
          <th>train_expected</th>
          <td>0.004091</td>
          <td>1.000000</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    df = pandas.DataFrame()
    df["test_predicted"] = clr.predict(X_test)
    df["test_expected"] = y_test
    df.corr()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>test_predicted</th>
          <th>test_expected</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>test_predicted</th>
          <td>1.00000</td>
          <td>-0.00921</td>
        </tr>
        <tr>
          <th>test_expected</th>
          <td>-0.00921</td>
          <td>1.00000</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    df.plot(x ="test_expected", y="test_predicted", kind="scatter");



.. image:: online_news_popylarity_38_0.png


`GradientBoostingRegressor <http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html>`__

.. code:: ipython3

    from sklearn.ensemble import GradientBoostingRegressor
    est = GradientBoostingRegressor(min_samples_leaf=20, n_estimators=50, min_weight_fraction_leaf=0.01, min_samples_split=10)
    est.fit(X_train, y_train)




.. parsed-literal::
    GradientBoostingRegressor(min_samples_leaf=20, min_samples_split=10,
                              min_weight_fraction_leaf=0.01, n_estimators=50)



.. code:: ipython3

    tpredicted = est.predict(X_train)
    df = pandas.DataFrame()
    df["train_predicted"] = tpredicted
    df["train_expected"] = y_train
    df.corr()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>train_predicted</th>
          <th>train_expected</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>train_predicted</th>
          <td>1.000000</td>
          <td>0.008707</td>
        </tr>
        <tr>
          <th>train_expected</th>
          <td>0.008707</td>
          <td>1.000000</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    df = pandas.DataFrame()
    df["train_predicted"] = est.predict(X_train)
    df["train_expected"] = y_train
    df.corr()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>train_predicted</th>
          <th>train_expected</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>train_predicted</th>
          <td>1.000000</td>
          <td>0.008707</td>
        </tr>
        <tr>
          <th>train_expected</th>
          <td>0.008707</td>
          <td>1.000000</td>
        </tr>
      </tbody>
    </table>
    </div>



`XGBRegressor <https://github.com/dmlc/xgboost/blob/master/demo/guide-python/sklearn_examples.py>`__

.. code:: ipython3

    import xgboost
    clxg = xgboost.XGBRegressor(max_depth=10, learning_rate=0.3, n_estimators=50)
    clxg.fit(X_train, y_train)




.. parsed-literal::
    XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,
                 colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,
                 importance_type='gain', interaction_constraints='',
                 learning_rate=0.3, max_delta_step=0, max_depth=10,
                 min_child_weight=1, missing=nan, monotone_constraints='()',
                 n_estimators=50, n_jobs=0, num_parallel_tree=1, random_state=0,
                 reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,
                 tree_method='exact', validate_parameters=1, verbosity=None)



.. code:: ipython3

    xgpredicted = clxg.predict(X_train)
    df = pandas.DataFrame()
    df["train_predicted"] = xgpredicted
    df["train_expected"] = y_train
    df.corr()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>train_predicted</th>
          <th>train_expected</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>train_predicted</th>
          <td>1.000000</td>
          <td>0.000811</td>
        </tr>
        <tr>
          <th>train_expected</th>
          <td>0.000811</td>
          <td>1.000000</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    # trop long
    #from sklearn import tree
    #from sklearn.ensemble import AdaBoostRegressor
    #clfr = tree.DecisionTreeRegressor(min_samples_leaf=10, min_samples_split=10)
    #clf2 = AdaBoostRegressor(clfr, n_estimators=800, learning_rate=0.5)
    #clf2.fit(X_train, y_train)

t-SNE
-----

`Comparison of Manifold Learning
methods <http://scikit-learn.org/stable/auto_examples/manifold/plot_compare_methods.html>`__,
`t-SNE <http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html>`__,
`t-distributed Stochastic Neighbor Embedding
(t-SNE) <http://scikit-learn.org/stable/modules/manifold.html#t-sne>`__

.. code:: ipython3

    from sklearn.model_selection import train_test_split
    X_1, X_2, y_1, y_2 = train_test_split(X_train.reset_index(drop=True), 
                            y_train.reset_index(drop=True), test_size=0.2, random_state=42)
    X_1.shape, X_2.shape




.. parsed-literal::
    ((23785, 59), (5947, 59))



.. code:: ipython3

    from sklearn.manifold import TSNE
    model = TSNE(n_components=2, random_state=0)
    model




.. parsed-literal::
    TSNE(random_state=0)



.. code:: ipython3

    W_2 = model.fit_transform(X_2)

.. code:: ipython3

    i_2 = y_2.astype(int)

.. code:: ipython3

    W_2.shape, X_2.shape, y_2.shape, i_2.shape




.. parsed-literal::
    ((5947, 2), (5947, 59), (5947,), (5947,))



.. code:: ipython3

    mini, maxi = min(i_2), max(i_2)+1

.. code:: ipython3

    import matplotlib.pyplot as plt
    f, ax = plt.subplots()
    
    for i in range(mini, maxi):
        ind = numpy.array(numpy.where(i_2==i)).T
        print(i, ind.shape)
        if i in(6,7,8):
            continue
        m = "o" if i <= 9 else "o"
        r = 1.0*i / maxi
        ax.plot(W_2[ind,0], W_2[ind,1], m, color =(r, 1-r, 0.0), label=str(i))
    ax.legend()
    ax;


.. parsed-literal::
    0 (1, 1)
    1 (1, 1)
    2 (0, 1)
    3 (3, 1)
    4 (16, 1)
    5 (38, 1)
    6 (1821, 1)
    7 (2646, 1)
    8 (978, 1)
    9 (321, 1)
    10 (96, 1)
    11 (24, 1)
    12 (2, 1)



.. image:: online_news_popylarity_54_1.png