.. _firststepsrst:

=================================
First steps with pandas_streaming
=================================


.. only:: html

    **Links:** :download:`notebook <first_steps.ipynb>`, :downloadlink:`html <first_steps2html.html>`, :download:`PDF <first_steps.pdf>`, :download:`python <first_steps.py>`, :downloadlink:`slides <first_steps.slides.html>`, :githublink:`GitHub|_doc/notebooks/first_steps.ipynb|*`


A few difference between `pandas <http://pandas.pydata.org/>`__ and
*pandas_streaming*.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





pandas to pandas_streaming
--------------------------

.. code:: ipython3

    from pandas import DataFrame
    df = DataFrame(data=dict(X=[4.5, 6, 7], Y=["a", "b", "c"]))
    df






.. raw:: html

    <div>
    <style>
        .dataframe thead tr:only-child th {
            text-align: right;
        }

        .dataframe thead th {
            text-align: left;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X</th>
          <th>Y</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>4.5</td>
          <td>a</td>
        </tr>
        <tr>
          <th>1</th>
          <td>6.0</td>
          <td>b</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7.0</td>
          <td>c</td>
        </tr>
      </tbody>
    </table>
    </div>



We create a streaming dataframe:

.. code:: ipython3

    from pandas_streaming.df import StreamingDataFrame
    sdf = StreamingDataFrame.read_df(df)
    sdf




.. parsed-literal::
    <pandas_streaming.df.dataframe.StreamingDataFrame at 0x15c2c606160>



.. code:: ipython3

    sdf.to_dataframe()






.. raw:: html

    <div>
    <style>
        .dataframe thead tr:only-child th {
            text-align: right;
        }

        .dataframe thead th {
            text-align: left;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X</th>
          <th>Y</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>4.5</td>
          <td>a</td>
        </tr>
        <tr>
          <th>1</th>
          <td>6.0</td>
          <td>b</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7.0</td>
          <td>c</td>
        </tr>
      </tbody>
    </table>
    </div>



Internally, StreamingDataFrame implements an iterator on dataframes and
then tries to replicate the same interface as
`pandas.DataFrame <https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html>`__
possibly wherever it is possible to manipulate data without loading
everything into memory.

.. code:: ipython3

    sdf2 = sdf.concat(sdf)
    sdf2.to_dataframe()






.. raw:: html

    <div>
    <style>
        .dataframe thead tr:only-child th {
            text-align: right;
        }

        .dataframe thead th {
            text-align: left;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X</th>
          <th>Y</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>4.5</td>
          <td>a</td>
        </tr>
        <tr>
          <th>1</th>
          <td>6.0</td>
          <td>b</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7.0</td>
          <td>c</td>
        </tr>
        <tr>
          <th>0</th>
          <td>4.5</td>
          <td>a</td>
        </tr>
        <tr>
          <th>1</th>
          <td>6.0</td>
          <td>b</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7.0</td>
          <td>c</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    m = DataFrame(dict(Y=["a", "b"], Z=[10, 20]))
    m






.. raw:: html

    <div>
    <style>
        .dataframe thead tr:only-child th {
            text-align: right;
        }

        .dataframe thead th {
            text-align: left;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Y</th>
          <th>Z</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>a</td>
          <td>10</td>
        </tr>
        <tr>
          <th>1</th>
          <td>b</td>
          <td>20</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    sdf3 = sdf2.merge(m, left_on="Y", right_on="Y", how="outer")
    sdf3.to_dataframe()






.. raw:: html

    <div>
    <style>
        .dataframe thead tr:only-child th {
            text-align: right;
        }

        .dataframe thead th {
            text-align: left;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X</th>
          <th>Y</th>
          <th>Z</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>4.5</td>
          <td>a</td>
          <td>10.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>6.0</td>
          <td>b</td>
          <td>20.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7.0</td>
          <td>c</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>0</th>
          <td>4.5</td>
          <td>a</td>
          <td>10.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>6.0</td>
          <td>b</td>
          <td>20.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>7.0</td>
          <td>c</td>
          <td>NaN</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    sdf2.to_dataframe().merge(m, left_on="Y", right_on="Y", how="outer")






.. raw:: html

    <div>
    <style>
        .dataframe thead tr:only-child th {
            text-align: right;
        }

        .dataframe thead th {
            text-align: left;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X</th>
          <th>Y</th>
          <th>Z</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>4.5</td>
          <td>a</td>
          <td>10.0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4.5</td>
          <td>a</td>
          <td>10.0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>6.0</td>
          <td>b</td>
          <td>20.0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>6.0</td>
          <td>b</td>
          <td>20.0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>7.0</td>
          <td>c</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>5</th>
          <td>7.0</td>
          <td>c</td>
          <td>NaN</td>
        </tr>
      </tbody>
    </table>
    </div>



The order might be different.

.. code:: ipython3

    sdftr, sdfte = sdf2.train_test_split(test_size=0.5)
    sdfte.head()






.. raw:: html

    <div>
    <style>
        .dataframe thead tr:only-child th {
            text-align: right;
        }

        .dataframe thead th {
            text-align: left;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X</th>
          <th>Y</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>4.5</td>
          <td>a</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4.5</td>
          <td>a</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    sdftr.head()






.. raw:: html

    <div>
    <style>
        .dataframe thead tr:only-child th {
            text-align: right;
        }

        .dataframe thead th {
            text-align: left;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>X</th>
          <th>Y</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>6.0</td>
          <td>b</td>
        </tr>
        <tr>
          <th>1</th>
          <td>7.0</td>
          <td>c</td>
        </tr>
        <tr>
          <th>2</th>
          <td>6.0</td>
          <td>b</td>
        </tr>
        <tr>
          <th>0</th>
          <td>7.0</td>
          <td>c</td>
        </tr>
      </tbody>
    </table>
    </div>



split a big file
----------------

.. code:: ipython3

    sdf2.to_csv("example.txt")




.. parsed-literal::
    'example.txt'



.. code:: ipython3

    new_sdf = StreamingDataFrame.read_csv("example.txt")
    new_sdf.train_test_split("example.{}.txt", streaming=False)




.. parsed-literal::
    ['example.train.txt', 'example.test.txt']



.. code:: ipython3

    import glob
    glob.glob("ex*.txt")




.. parsed-literal::
    ['example.test.txt', 'example.train.txt', 'example.txt']