.. _citybikeviewsrst:

===============
City Bike Views
===============


.. only:: html

    **Links:** :download:`notebook <city_bike_views.ipynb>`, :downloadlink:`html <city_bike_views2html.html>`, :download:`PDF <city_bike_views.pdf>`, :download:`python <city_bike_views.py>`, :downloadlink:`slides <city_bike_views.slides.html>`, :githublink:`GitHub|_doc/notebooks/challenges/city_bike/city_bike_views.ipynb|*`


Based on the data available at `Divvy
Data <https://www.divvybikes.com/system-data>`__, some ways to look at
the data.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





.. code:: ipython3

    %matplotlib inline

The data
--------

`Divvy Data <https://www.divvybikes.com/system-data>`__ publishes a
sample of the data.

.. code:: ipython3

    from pyensae.datasource import download_data
    file = download_data("Divvy_Trips_2016_Q3Q4.zip", url="https://s3.amazonaws.com/divvy-data/tripdata/")

.. code:: ipython3

    import pandas
    stations = pandas.read_csv("Divvy_Stations_2016_Q3.csv")
    bikes = pandas.concat([pandas.read_csv("Divvy_Trips_2016_Q3.csv"),
                           pandas.read_csv("Divvy_Trips_2016_Q4.csv")])

.. code:: ipython3

    bikes.head()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>trip_id</th>
          <th>starttime</th>
          <th>stoptime</th>
          <th>bikeid</th>
          <th>tripduration</th>
          <th>from_station_id</th>
          <th>from_station_name</th>
          <th>to_station_id</th>
          <th>to_station_name</th>
          <th>usertype</th>
          <th>gender</th>
          <th>birthyear</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>12150160</td>
          <td>9/30/2016 23:59:58</td>
          <td>10/1/2016 00:04:03</td>
          <td>4959</td>
          <td>245</td>
          <td>69</td>
          <td>Damen Ave &amp; Pierce Ave</td>
          <td>17</td>
          <td>Wood St &amp; Division St</td>
          <td>Subscriber</td>
          <td>Male</td>
          <td>1988.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>12150159</td>
          <td>9/30/2016 23:59:58</td>
          <td>10/1/2016 00:04:09</td>
          <td>2589</td>
          <td>251</td>
          <td>383</td>
          <td>Ashland Ave &amp; Harrison St</td>
          <td>320</td>
          <td>Loomis St &amp; Lexington St</td>
          <td>Subscriber</td>
          <td>Female</td>
          <td>1990.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>12150158</td>
          <td>9/30/2016 23:59:51</td>
          <td>10/1/2016 00:24:51</td>
          <td>3656</td>
          <td>1500</td>
          <td>302</td>
          <td>Sheffield Ave &amp; Wrightwood Ave</td>
          <td>334</td>
          <td>Lake Shore Dr &amp; Belmont Ave</td>
          <td>Customer</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>3</th>
          <td>12150157</td>
          <td>9/30/2016 23:59:51</td>
          <td>10/1/2016 00:03:56</td>
          <td>3570</td>
          <td>245</td>
          <td>475</td>
          <td>Washtenaw Ave &amp; Lawrence Ave</td>
          <td>471</td>
          <td>Francisco Ave &amp; Foster Ave</td>
          <td>Subscriber</td>
          <td>Female</td>
          <td>1988.0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>12150156</td>
          <td>9/30/2016 23:59:32</td>
          <td>10/1/2016 00:26:50</td>
          <td>3158</td>
          <td>1638</td>
          <td>302</td>
          <td>Sheffield Ave &amp; Wrightwood Ave</td>
          <td>492</td>
          <td>Leavitt St &amp; Addison St</td>
          <td>Customer</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
      </tbody>
    </table>
    </div>



About age
---------

.. code:: ipython3

    from datetime import datetime, time
    df = bikes
    df["dtstart"] = pandas.to_datetime(df.starttime, infer_datetime_format=True)
    df["dtstop"] = pandas.to_datetime(df.stoptime, infer_datetime_format=True)
    df["stopday"] = df.dtstop.apply(lambda r: datetime(r.year, r.month, r.day))
    df["stoptime"] = df.dtstop.apply(lambda r: time(r.hour, r.minute, 0))
    df["stoptime10"] = df.dtstop.apply(lambda r: time(r.hour, (r.minute // 10)*10, 0))  # every 10 minutes
    df['stopweekday'] = df['dtstop'].dt.dayofweek

.. code:: ipython3

    df['duration'] = df["dtstop"] -  df["dtstart"]
    df["age"] = - df["birthyear"] + 2016
    df['duration_sec'] = df['duration'].apply(lambda x: x.total_seconds())

.. code:: ipython3

    df["stoptime_sec"] = df.dtstop.apply(lambda r: r.hour * 60 + r.minute)

.. code:: ipython3

    df.describe().T






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>count</th>
          <th>mean</th>
          <th>std</th>
          <th>min</th>
          <th>25%</th>
          <th>50%</th>
          <th>75%</th>
          <th>max</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>trip_id</th>
          <td>2.12564e+06</td>
          <td>1.16993e+07</td>
          <td>731388</td>
          <td>1.04267e+07</td>
          <td>1.10663e+07</td>
          <td>1.17015e+07</td>
          <td>1.23313e+07</td>
          <td>1.29792e+07</td>
        </tr>
        <tr>
          <th>bikeid</th>
          <td>2.12564e+06</td>
          <td>3251.98</td>
          <td>1730.44</td>
          <td>1</td>
          <td>1755</td>
          <td>3446</td>
          <td>4802</td>
          <td>5920</td>
        </tr>
        <tr>
          <th>tripduration</th>
          <td>2.12564e+06</td>
          <td>1008.55</td>
          <td>1816.1</td>
          <td>60</td>
          <td>416</td>
          <td>716</td>
          <td>1195</td>
          <td>86365</td>
        </tr>
        <tr>
          <th>from_station_id</th>
          <td>2.12564e+06</td>
          <td>179.916</td>
          <td>130.524</td>
          <td>2</td>
          <td>75</td>
          <td>157</td>
          <td>268</td>
          <td>620</td>
        </tr>
        <tr>
          <th>to_station_id</th>
          <td>2.12564e+06</td>
          <td>180.352</td>
          <td>130.488</td>
          <td>2</td>
          <td>75</td>
          <td>157</td>
          <td>272</td>
          <td>620</td>
        </tr>
        <tr>
          <th>birthyear</th>
          <td>1.59034e+06</td>
          <td>1980.79</td>
          <td>10.754</td>
          <td>1899</td>
          <td>1975</td>
          <td>1984</td>
          <td>1989</td>
          <td>2000</td>
        </tr>
        <tr>
          <th>stopweekday</th>
          <td>2.12564e+06</td>
          <td>2.95275</td>
          <td>2.02016</td>
          <td>0</td>
          <td>1</td>
          <td>3</td>
          <td>5</td>
          <td>6</td>
        </tr>
        <tr>
          <th>duration</th>
          <td>2125643</td>
          <td>0 days 00:16:48.183487</td>
          <td>0 days 00:30:15.597468</td>
          <td>0 days 00:00:59</td>
          <td>0 days 00:06:56</td>
          <td>0 days 00:11:56</td>
          <td>0 days 00:19:55</td>
          <td>0 days 23:59:24</td>
        </tr>
        <tr>
          <th>age</th>
          <td>1.59034e+06</td>
          <td>35.2129</td>
          <td>10.754</td>
          <td>16</td>
          <td>27</td>
          <td>32</td>
          <td>41</td>
          <td>117</td>
        </tr>
        <tr>
          <th>duration_sec</th>
          <td>2.12564e+06</td>
          <td>1008.18</td>
          <td>1815.6</td>
          <td>59</td>
          <td>416</td>
          <td>716</td>
          <td>1195</td>
          <td>86364</td>
        </tr>
        <tr>
          <th>stoptime_sec</th>
          <td>2.12564e+06</td>
          <td>869.324</td>
          <td>284.732</td>
          <td>0</td>
          <td>652</td>
          <td>919</td>
          <td>1080</td>
          <td>1439</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    df.shape




.. parsed-literal::
    (2125643, 21)



We take a random sample.

.. code:: ipython3

    import random
    ens = pandas.Series([random.randint(0,99) for i in range(df.shape[0])])
    sample = df[ens==0]


.. parsed-literal::
    c:\python370_x64\lib\site-packages\ipykernel_launcher.py:3: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
      This is separate from the ipykernel package so we can avoid doing imports until


.. code:: ipython3

    sample.shape
    sample = sample[(sample.age < 100) & (sample.duration_sec < 3600)]

.. code:: ipython3

    import numpy as np
    import pandas as pd
    import seaborn as sns
    sns.set(style="white")
    g = sns.jointplot(sample.age, sample.duration_sec, kind="kde", size=7, space=0)


.. parsed-literal::
    c:\python370_x64\lib\site-packages\seaborn\axisgrid.py:2262: UserWarning: The `size` paramter has been renamed to `height`; please update your code.
      warnings.warn(msg, UserWarning)
    c:\python370_x64\lib\site-packages\scipy\stats\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
      return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval



.. image:: city_bike_views_16_1.png


.. code:: ipython3

    import numpy as np
    import pandas as pd
    import seaborn as sns
    sns.set(style="white")
    g = sns.jointplot(sample.age, sample.stoptime_sec, kind="kde", size=7, space=0)


.. parsed-literal::
    c:\python370_x64\lib\site-packages\seaborn\axisgrid.py:2262: UserWarning: The `size` paramter has been renamed to `height`; please update your code.
      warnings.warn(msg, UserWarning)
    c:\python370_x64\lib\site-packages\scipy\stats\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
      return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval



.. image:: city_bike_views_17_1.png


The duration seems correlated to the age. Let’s see. Younger people
during the weekend are more active and bike longer.

.. code:: ipython3

    import matplotlib.pyplot as plt
    fig, ax = plt.subplots(1,2, figsize=(14,4))
    sns.boxplot(x="stopweekday", y="age", data=df[df.age < 100], color="c", ax=ax[0])
    sns.boxplot(x="stopweekday", y="duration_sec", data=df[df.duration_sec < 3600], color="c", ax=ax[1]);



.. image:: city_bike_views_19_0.png


However, linear correlations are not so great.

.. code:: ipython3

    df.corr()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>trip_id</th>
          <th>bikeid</th>
          <th>tripduration</th>
          <th>from_station_id</th>
          <th>to_station_id</th>
          <th>birthyear</th>
          <th>stopweekday</th>
          <th>age</th>
          <th>duration_sec</th>
          <th>stoptime_sec</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>trip_id</th>
          <td>1.000000</td>
          <td>-0.025039</td>
          <td>-0.071566</td>
          <td>0.008129</td>
          <td>0.005505</td>
          <td>-0.035032</td>
          <td>-0.067585</td>
          <td>0.035032</td>
          <td>-0.071514</td>
          <td>-0.063709</td>
        </tr>
        <tr>
          <th>bikeid</th>
          <td>-0.025039</td>
          <td>1.000000</td>
          <td>0.001088</td>
          <td>0.009959</td>
          <td>0.009538</td>
          <td>-0.010901</td>
          <td>0.000779</td>
          <td>0.010901</td>
          <td>0.001090</td>
          <td>-0.007245</td>
        </tr>
        <tr>
          <th>tripduration</th>
          <td>-0.071566</td>
          <td>0.001088</td>
          <td>1.000000</td>
          <td>-0.008972</td>
          <td>-0.004730</td>
          <td>-0.009788</td>
          <td>0.069885</td>
          <td>0.009788</td>
          <td>0.999969</td>
          <td>0.034435</td>
        </tr>
        <tr>
          <th>from_station_id</th>
          <td>0.008129</td>
          <td>0.009959</td>
          <td>-0.008972</td>
          <td>1.000000</td>
          <td>0.386314</td>
          <td>0.019982</td>
          <td>0.019426</td>
          <td>-0.019982</td>
          <td>-0.008970</td>
          <td>-0.016486</td>
        </tr>
        <tr>
          <th>to_station_id</th>
          <td>0.005505</td>
          <td>0.009538</td>
          <td>-0.004730</td>
          <td>0.386314</td>
          <td>1.000000</td>
          <td>0.021198</td>
          <td>0.011177</td>
          <td>-0.021198</td>
          <td>-0.004734</td>
          <td>0.061833</td>
        </tr>
        <tr>
          <th>birthyear</th>
          <td>-0.035032</td>
          <td>-0.010901</td>
          <td>-0.009788</td>
          <td>0.019982</td>
          <td>0.021198</td>
          <td>1.000000</td>
          <td>0.057081</td>
          <td>-1.000000</td>
          <td>-0.009808</td>
          <td>0.085929</td>
        </tr>
        <tr>
          <th>stopweekday</th>
          <td>-0.067585</td>
          <td>0.000779</td>
          <td>0.069885</td>
          <td>0.019426</td>
          <td>0.011177</td>
          <td>0.057081</td>
          <td>1.000000</td>
          <td>-0.057081</td>
          <td>0.069848</td>
          <td>0.017866</td>
        </tr>
        <tr>
          <th>age</th>
          <td>0.035032</td>
          <td>0.010901</td>
          <td>0.009788</td>
          <td>-0.019982</td>
          <td>-0.021198</td>
          <td>-1.000000</td>
          <td>-0.057081</td>
          <td>1.000000</td>
          <td>0.009808</td>
          <td>-0.085929</td>
        </tr>
        <tr>
          <th>duration_sec</th>
          <td>-0.071514</td>
          <td>0.001090</td>
          <td>0.999969</td>
          <td>-0.008970</td>
          <td>-0.004734</td>
          <td>-0.009808</td>
          <td>0.069848</td>
          <td>0.009808</td>
          <td>1.000000</td>
          <td>0.034514</td>
        </tr>
        <tr>
          <th>stoptime_sec</th>
          <td>-0.063709</td>
          <td>-0.007245</td>
          <td>0.034435</td>
          <td>-0.016486</td>
          <td>0.061833</td>
          <td>0.085929</td>
          <td>0.017866</td>
          <td>-0.085929</td>
          <td>0.034514</td>
          <td>1.000000</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    fig, ax = plt.subplots(1,2, figsize=(14,4))
    sns.violinplot(x="stopweekday", y="duration_sec", hue="gender", data=sample, split=True, inner="quart", ax=ax[0])
    sns.violinplot(x="stopweekday", y="age", hue="gender", data=sample, split=True, inner="quart", ax=ax[1])
    sns.despine(left=True);



.. image:: city_bike_views_22_0.png


.. code:: ipython3

    fig, ax = plt.subplots(1,2, figsize=(14,4))
    sns.violinplot(x="stopweekday", y="stoptime_sec", hue="gender", data=sample, split=True, inner="quart", ax=ax[0])
    sns.violinplot(x="usertype", y="stoptime_sec", hue="gender", data=sample, split=True, inner="quart", ax=ax[1])
    sns.despine(left=True);



.. image:: city_bike_views_23_0.png


.. code:: ipython3

    fig, ax = plt.subplots(1,2, figsize=(14,4))
    sns.violinplot(x="usertype", y="duration_sec", hue="gender", data=sample, split=True, inner="quart", ax=ax[0])
    sns.violinplot(x="usertype", y="age", hue="gender", data=sample, split=True, inner="quart", ax=ax[1])
    sns.despine(left=True);



.. image:: city_bike_views_24_0.png


Non-linear correlations
-----------------------

We apply the following `Corrélations non
linéaires <http://www.xavierdupre.fr/app/mlstatpy/helpsphinx/notebooks/correlation_non_lineaire.html#overfitting>`__.

.. code:: ipython3

    sample2 = sample.copy()
    sample2["age_inv"] = sample2.age ** -1
    sample2["gender_num"] = sample2.gender.apply(lambda x: (1 if x == "Male" else 0))
    sample2["usertype_num"] = sample2.usertype.apply(lambda x: (1 if x == "Subscriber" else 0))
    sample2.corr()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>trip_id</th>
          <th>bikeid</th>
          <th>tripduration</th>
          <th>from_station_id</th>
          <th>to_station_id</th>
          <th>birthyear</th>
          <th>stopweekday</th>
          <th>age</th>
          <th>duration_sec</th>
          <th>stoptime_sec</th>
          <th>age_inv</th>
          <th>gender_num</th>
          <th>usertype_num</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>trip_id</th>
          <td>1.000000</td>
          <td>-0.020790</td>
          <td>-0.090476</td>
          <td>-0.000107</td>
          <td>-0.013945</td>
          <td>-0.032304</td>
          <td>-0.044524</td>
          <td>0.032304</td>
          <td>-0.089994</td>
          <td>-0.050106</td>
          <td>-0.027614</td>
          <td>0.058240</td>
          <td>0.014886</td>
        </tr>
        <tr>
          <th>bikeid</th>
          <td>-0.020790</td>
          <td>1.000000</td>
          <td>0.011953</td>
          <td>0.009183</td>
          <td>0.013776</td>
          <td>-0.011606</td>
          <td>-0.001548</td>
          <td>0.011606</td>
          <td>0.011944</td>
          <td>-0.004755</td>
          <td>-0.009776</td>
          <td>0.031241</td>
          <td>-0.000271</td>
        </tr>
        <tr>
          <th>tripduration</th>
          <td>-0.090476</td>
          <td>0.011953</td>
          <td>1.000000</td>
          <td>0.044958</td>
          <td>0.050456</td>
          <td>-0.005926</td>
          <td>0.063156</td>
          <td>0.005926</td>
          <td>0.999999</td>
          <td>0.073529</td>
          <td>-0.016003</td>
          <td>-0.108492</td>
          <td>-0.008061</td>
        </tr>
        <tr>
          <th>from_station_id</th>
          <td>-0.000107</td>
          <td>0.009183</td>
          <td>0.044958</td>
          <td>1.000000</td>
          <td>0.381516</td>
          <td>0.021112</td>
          <td>0.051470</td>
          <td>-0.021112</td>
          <td>0.044957</td>
          <td>-0.010079</td>
          <td>0.029148</td>
          <td>-0.027132</td>
          <td>0.019338</td>
        </tr>
        <tr>
          <th>to_station_id</th>
          <td>-0.013945</td>
          <td>0.013776</td>
          <td>0.050456</td>
          <td>0.381516</td>
          <td>1.000000</td>
          <td>0.027436</td>
          <td>0.049334</td>
          <td>-0.027436</td>
          <td>0.050440</td>
          <td>0.089843</td>
          <td>0.033983</td>
          <td>-0.031992</td>
          <td>0.014013</td>
        </tr>
        <tr>
          <th>birthyear</th>
          <td>-0.032304</td>
          <td>-0.011606</td>
          <td>-0.005926</td>
          <td>0.021112</td>
          <td>0.027436</td>
          <td>1.000000</td>
          <td>0.047890</td>
          <td>-1.000000</td>
          <td>-0.005937</td>
          <td>0.090793</td>
          <td>0.951178</td>
          <td>-0.080894</td>
          <td>0.008748</td>
        </tr>
        <tr>
          <th>stopweekday</th>
          <td>-0.044524</td>
          <td>-0.001548</td>
          <td>0.063156</td>
          <td>0.051470</td>
          <td>0.049334</td>
          <td>0.047890</td>
          <td>1.000000</td>
          <td>-0.047890</td>
          <td>0.063146</td>
          <td>-0.005645</td>
          <td>0.052167</td>
          <td>-0.043754</td>
          <td>-0.009274</td>
        </tr>
        <tr>
          <th>age</th>
          <td>0.032304</td>
          <td>0.011606</td>
          <td>0.005926</td>
          <td>-0.021112</td>
          <td>-0.027436</td>
          <td>-1.000000</td>
          <td>-0.047890</td>
          <td>1.000000</td>
          <td>0.005937</td>
          <td>-0.090793</td>
          <td>-0.951178</td>
          <td>0.080894</td>
          <td>-0.008748</td>
        </tr>
        <tr>
          <th>duration_sec</th>
          <td>-0.089994</td>
          <td>0.011944</td>
          <td>0.999999</td>
          <td>0.044957</td>
          <td>0.050440</td>
          <td>-0.005937</td>
          <td>0.063146</td>
          <td>0.005937</td>
          <td>1.000000</td>
          <td>0.073511</td>
          <td>-0.016010</td>
          <td>-0.108476</td>
          <td>-0.008063</td>
        </tr>
        <tr>
          <th>stoptime_sec</th>
          <td>-0.050106</td>
          <td>-0.004755</td>
          <td>0.073529</td>
          <td>-0.010079</td>
          <td>0.089843</td>
          <td>0.090793</td>
          <td>-0.005645</td>
          <td>-0.090793</td>
          <td>0.073511</td>
          <td>1.000000</td>
          <td>0.091370</td>
          <td>-0.003124</td>
          <td>-0.005995</td>
        </tr>
        <tr>
          <th>age_inv</th>
          <td>-0.027614</td>
          <td>-0.009776</td>
          <td>-0.016003</td>
          <td>0.029148</td>
          <td>0.033983</td>
          <td>0.951178</td>
          <td>0.052167</td>
          <td>-0.951178</td>
          <td>-0.016010</td>
          <td>0.091370</td>
          <td>1.000000</td>
          <td>-0.088716</td>
          <td>0.009361</td>
        </tr>
        <tr>
          <th>gender_num</th>
          <td>0.058240</td>
          <td>0.031241</td>
          <td>-0.108492</td>
          <td>-0.027132</td>
          <td>-0.031992</td>
          <td>-0.080894</td>
          <td>-0.043754</td>
          <td>0.080894</td>
          <td>-0.108476</td>
          <td>-0.003124</td>
          <td>-0.088716</td>
          <td>1.000000</td>
          <td>-0.009875</td>
        </tr>
        <tr>
          <th>usertype_num</th>
          <td>0.014886</td>
          <td>-0.000271</td>
          <td>-0.008061</td>
          <td>0.019338</td>
          <td>0.014013</td>
          <td>0.008748</td>
          <td>-0.009274</td>
          <td>-0.008748</td>
          <td>-0.008063</td>
          <td>-0.005995</td>
          <td>0.009361</td>
          <td>-0.009875</td>
          <td>1.000000</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    from sklearn.model_selection import train_test_split
    from sklearn.preprocessing import scale
    import numpy
    
    def correlation_cross_val(df, model, draws=5, **params):
        cor = df.corr()    
        df = scale(df[cor.columns])
        for i in range(cor.shape[0]):
            xi = df[:, i:i+1]
            for j in range(cor.shape[1]):
                xj = df[:, j]
                mem = []
                for k in range(0, draws):
                    xi_train, xi_test, xj_train, xj_test = train_test_split(xi, xj, train_size=0.5)
                    mod = model(**params)
                    mod.fit(xi_train, xj_train)
                    v = mod.predict(xi_test)
                    c = (1 - numpy.var(v - xj_test))
                    mem.append(max(c, 0) **0.5)
                cor.iloc[i,j] = sum(mem) / len(mem)
        return cor
    
    from sklearn.tree import DecisionTreeRegressor
    cor = correlation_cross_val(sample2, DecisionTreeRegressor, draws=20)
    cor


.. parsed-literal::
    c:\python370_x64\lib\site-packages\sklearn\model_selection\_split.py:2026: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.
      FutureWarning)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>trip_id</th>
          <th>bikeid</th>
          <th>tripduration</th>
          <th>from_station_id</th>
          <th>to_station_id</th>
          <th>birthyear</th>
          <th>stopweekday</th>
          <th>age</th>
          <th>duration_sec</th>
          <th>stoptime_sec</th>
          <th>age_inv</th>
          <th>gender_num</th>
          <th>usertype_num</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>trip_id</th>
          <td>1.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.990057</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.878676</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.015071</td>
        </tr>
        <tr>
          <th>bikeid</th>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.017745</td>
        </tr>
        <tr>
          <th>tripduration</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.999990</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.999990</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.099688</td>
        </tr>
        <tr>
          <th>from_station_id</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.054325</td>
          <td>0.999999</td>
          <td>0.536015</td>
          <td>0.008031</td>
          <td>0.000000</td>
          <td>0.006133</td>
          <td>0.044061</td>
          <td>0.012246</td>
          <td>0.028895</td>
          <td>0.000000</td>
          <td>0.172352</td>
        </tr>
        <tr>
          <th>to_station_id</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.109784</td>
          <td>0.532589</td>
          <td>0.999999</td>
          <td>0.026304</td>
          <td>0.000000</td>
          <td>0.024202</td>
          <td>0.105040</td>
          <td>0.248720</td>
          <td>0.019269</td>
          <td>0.000000</td>
          <td>0.153788</td>
        </tr>
        <tr>
          <th>birthyear</th>
          <td>0.011917</td>
          <td>0.005125</td>
          <td>0.058987</td>
          <td>0.026033</td>
          <td>0.034188</td>
          <td>0.999965</td>
          <td>0.006681</td>
          <td>0.999942</td>
          <td>0.053868</td>
          <td>0.055993</td>
          <td>0.999996</td>
          <td>0.068499</td>
          <td>0.209815</td>
        </tr>
        <tr>
          <th>stopweekday</th>
          <td>0.042561</td>
          <td>0.020920</td>
          <td>0.079774</td>
          <td>0.065344</td>
          <td>0.045880</td>
          <td>0.085778</td>
          <td>1.000000</td>
          <td>0.055188</td>
          <td>0.074982</td>
          <td>0.033219</td>
          <td>0.095588</td>
          <td>0.054960</td>
          <td>0.291366</td>
        </tr>
        <tr>
          <th>age</th>
          <td>0.011934</td>
          <td>0.006577</td>
          <td>0.051631</td>
          <td>0.019733</td>
          <td>0.028614</td>
          <td>0.999909</td>
          <td>0.017197</td>
          <td>0.999943</td>
          <td>0.063061</td>
          <td>0.052308</td>
          <td>0.999997</td>
          <td>0.055599</td>
          <td>0.332540</td>
        </tr>
        <tr>
          <th>duration_sec</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.999991</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.999992</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.026094</td>
        </tr>
        <tr>
          <th>stoptime_sec</th>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>1.000000</td>
          <td>0.000000</td>
          <td>0.000000</td>
          <td>0.114911</td>
        </tr>
        <tr>
          <th>age_inv</th>
          <td>0.017233</td>
          <td>0.009412</td>
          <td>0.042053</td>
          <td>0.028141</td>
          <td>0.010642</td>
          <td>0.999918</td>
          <td>0.024621</td>
          <td>0.999912</td>
          <td>0.022328</td>
          <td>0.051536</td>
          <td>0.999997</td>
          <td>0.070596</td>
          <td>0.275990</td>
        </tr>
        <tr>
          <th>gender_num</th>
          <td>0.049358</td>
          <td>0.032177</td>
          <td>0.073034</td>
          <td>0.038476</td>
          <td>0.065741</td>
          <td>0.080659</td>
          <td>0.039153</td>
          <td>0.062295</td>
          <td>0.079421</td>
          <td>0.052172</td>
          <td>0.085579</td>
          <td>1.000000</td>
          <td>0.245688</td>
        </tr>
        <tr>
          <th>usertype_num</th>
          <td>0.032215</td>
          <td>0.041777</td>
          <td>0.063852</td>
          <td>0.039527</td>
          <td>0.032406</td>
          <td>0.038315</td>
          <td>0.037363</td>
          <td>0.055648</td>
          <td>0.059596</td>
          <td>0.037164</td>
          <td>0.039390</td>
          <td>0.035496</td>
          <td>1.000000</td>
        </tr>
      </tbody>
    </table>
    </div>



*from_station_id* and *start_station_id* seem related. Which means there
is frequent trip. Funny, the trip id can explain the stopping time… It
should be removed from any dataset.

.. code:: ipython3

    sample2.plot(x="trip_id", y="stoptime_sec", kind="scatter", figsize=(14,4));



.. image:: city_bike_views_29_0.png