.. _bigdatawaysrst:

===============
Séance big data
===============


.. only:: html

    **Links:** :download:`notebook <bigdata_ways.ipynb>`, :downloadlink:`html <bigdata_ways2html.html>`, :download:`PDF <bigdata_ways.pdf>`, :download:`python <bigdata_ways.py>`, :downloadlink:`slides <bigdata_ways.slides.html>`, :githublink:`GitHub|_doc/notebooks/sessions/bigdata_ways.ipynb|*`


Découverte de plusieurs façons de calculer des statistiques sur des
fichiers volumineux.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





.. code:: ipython3

    %matplotlib inline

Les données
-----------

On récupère un fichier de la
`DAMIR <https://www.data.gouv.fr/fr/datasets/open-damir-base-complete-sur-les-depenses-dassurance-maladie-inter-regimes/>`__.
Le fichier une fois décompressé fait plus de quatre Go. Vous pouvez le
télécharger directement depuis le site ou prendre une version de backup.
Vous trouvez d’autres données intéressantes sur le sujet sur le projet
`SGMAP-AGD/DAMIR <https://github.com/SGMAP-AGD/DAMIR>`__.

.. code:: ipython3

    import os
    filename = "bigdata/A201612_small.csv"
    # pour travailler avec un fichier plus gros (4 Go)
    # filename = "bigdata/A201612.csv"
    xlsfile = "bigdata/Lexique_open-DAMIR.xls"
    if not os.path.exists(filename):
        from pyensae.datasource import download_data
        if not os.path.exists("bigdata"):
            os.mkdir("bigdata")
        filenamegz = filename.replace(".csv", ".csv.gz")
        download_data(os.path.split(filenamegz)[-1], whereTo=".")
        download_data(os.path.split(xlsfile)[-1], whereTo="bigdata")
        if not os.path.exists(filename):
            filename = os.path.split(filename)[-1]

.. code:: ipython3

    os.stat(filename).st_size / 2**30




.. parsed-literal::
    0.015162252821028233



Il est impossible de le charger en mémoire en entier. On regarde les
premières lignes.

.. code:: ipython3

    with open(filename, "r") as f:
        for i, line in enumerate(f):
            print(line.strip("\r\n"))
            if i >= 5:
                break


.. parsed-literal::
    FLX_ANN_MOI;ORG_CLE_REG;AGE_BEN_SNDS;BEN_RES_REG;BEN_CMU_TOP;BEN_QLT_COD;BEN_SEX_COD;DDP_SPE_COD;ETE_CAT_SNDS;ETE_REG_COD;ETE_TYP_SNDS;ETP_REG_COD;ETP_CAT_SNDS;MDT_TYP_COD;MFT_COD;PRS_FJH_TYP;PRS_ACT_COG;PRS_ACT_NBR;PRS_ACT_QTE;PRS_DEP_MNT;PRS_PAI_MNT;PRS_REM_BSE;PRS_REM_MNT;FLT_ACT_COG;FLT_ACT_NBR;FLT_ACT_QTE;FLT_PAI_MNT;FLT_DEP_MNT;FLT_REM_MNT;SOI_ANN;SOI_MOI;ASU_NAT;ATT_NAT;CPL_COD;CPT_ENV_TYP;DRG_AFF_NAT;ETE_IND_TAA;EXO_MTF;MTM_NAT;PRS_NAT;PRS_PPU_SEC;PRS_REM_TAU;PRS_REM_TYP;PRS_PDS_QCP;EXE_INS_REG;PSE_ACT_SNDS;PSE_ACT_CAT;PSE_SPE_SNDS;PSE_STJ_SNDS;PRE_INS_REG;PSP_ACT_SNDS;PSP_ACT_CAT;PSP_SPE_SNDS;PSP_STJ_SNDS;TOP_PS5_TRG;
    201612;76;20;76;9;1;1;0;1101;76;1;76;1101;9;3;0;2;1;1;0;46;46;46;2;1;1;46;0;46;2016;12;40;0;0;9;99;1;99;9;1117;1;100;99;3;99;0;0;14;2;99;0;0;1;2;9;
    201612;28;30;28;9;1;2;0;9999;99;99;24;1102;9;99;0;1;5;1;0;.51;.51;.51;1;5;1;.51;0;.51;2016;12;10;0;0;9;99;9;99;9;3380;2;100;99;31;28;50;2;0;1;99;0;0;6;2;9;
    


On calcule le nombre de lignes :

.. code:: ipython3

    with open(filename, "r") as f:
        nb = 0
        for line in f:
            nb += 1
    nb




.. parsed-literal::
    199999



On peut aussi utiliser
`pandas.read_csv <http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html>`__
pour regarder les premières lignes.

.. code:: ipython3

    import time
    import pandas
    begin = time.perf_counter()
    df = pandas.read_csv(filename, sep=";", nrows=100)
    print(time.perf_counter() - begin)


.. parsed-literal::
    0.02099410000002422


.. code:: ipython3

    df.head()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>FLX_ANN_MOI</th>
          <th>ORG_CLE_REG</th>
          <th>AGE_BEN_SNDS</th>
          <th>BEN_RES_REG</th>
          <th>BEN_CMU_TOP</th>
          <th>BEN_QLT_COD</th>
          <th>BEN_SEX_COD</th>
          <th>DDP_SPE_COD</th>
          <th>ETE_CAT_SNDS</th>
          <th>ETE_REG_COD</th>
          <th>...</th>
          <th>PSE_ACT_CAT</th>
          <th>PSE_SPE_SNDS</th>
          <th>PSE_STJ_SNDS</th>
          <th>PRE_INS_REG</th>
          <th>PSP_ACT_SNDS</th>
          <th>PSP_ACT_CAT</th>
          <th>PSP_SPE_SNDS</th>
          <th>PSP_STJ_SNDS</th>
          <th>TOP_PS5_TRG</th>
          <th>Unnamed: 55</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>201612</td>
          <td>76</td>
          <td>20</td>
          <td>76</td>
          <td>9</td>
          <td>1</td>
          <td>1</td>
          <td>0</td>
          <td>1101</td>
          <td>76</td>
          <td>...</td>
          <td>0</td>
          <td>14</td>
          <td>2</td>
          <td>99</td>
          <td>0</td>
          <td>0</td>
          <td>1</td>
          <td>2</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>1</th>
          <td>201612</td>
          <td>28</td>
          <td>30</td>
          <td>28</td>
          <td>9</td>
          <td>1</td>
          <td>2</td>
          <td>0</td>
          <td>9999</td>
          <td>99</td>
          <td>...</td>
          <td>2</td>
          <td>0</td>
          <td>1</td>
          <td>99</td>
          <td>0</td>
          <td>0</td>
          <td>6</td>
          <td>2</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>2</th>
          <td>201612</td>
          <td>27</td>
          <td>60</td>
          <td>93</td>
          <td>9</td>
          <td>2</td>
          <td>1</td>
          <td>0</td>
          <td>9999</td>
          <td>99</td>
          <td>...</td>
          <td>2</td>
          <td>0</td>
          <td>9</td>
          <td>93</td>
          <td>0</td>
          <td>1</td>
          <td>1</td>
          <td>9</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>3</th>
          <td>201612</td>
          <td>32</td>
          <td>40</td>
          <td>32</td>
          <td>9</td>
          <td>2</td>
          <td>2</td>
          <td>0</td>
          <td>9999</td>
          <td>99</td>
          <td>...</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>32</td>
          <td>0</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>4</th>
          <td>201612</td>
          <td>32</td>
          <td>30</td>
          <td>32</td>
          <td>9</td>
          <td>1</td>
          <td>2</td>
          <td>0</td>
          <td>2206</td>
          <td>32</td>
          <td>...</td>
          <td>0</td>
          <td>0</td>
          <td>2</td>
          <td>99</td>
          <td>0</td>
          <td>0</td>
          <td>1</td>
          <td>2</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
      </tbody>
    </table>
    <p>5 rows × 56 columns</p>
    </div>



.. code:: ipython3

    df.head().T






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
          <th>3</th>
          <th>4</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>FLX_ANN_MOI</th>
          <td>201612.0</td>
          <td>201612.00</td>
          <td>201612.0</td>
          <td>201612.0</td>
          <td>201612.0</td>
        </tr>
        <tr>
          <th>ORG_CLE_REG</th>
          <td>76.0</td>
          <td>28.00</td>
          <td>27.0</td>
          <td>32.0</td>
          <td>32.0</td>
        </tr>
        <tr>
          <th>AGE_BEN_SNDS</th>
          <td>20.0</td>
          <td>30.00</td>
          <td>60.0</td>
          <td>40.0</td>
          <td>30.0</td>
        </tr>
        <tr>
          <th>BEN_RES_REG</th>
          <td>76.0</td>
          <td>28.00</td>
          <td>93.0</td>
          <td>32.0</td>
          <td>32.0</td>
        </tr>
        <tr>
          <th>BEN_CMU_TOP</th>
          <td>9.0</td>
          <td>9.00</td>
          <td>9.0</td>
          <td>9.0</td>
          <td>9.0</td>
        </tr>
        <tr>
          <th>BEN_QLT_COD</th>
          <td>1.0</td>
          <td>1.00</td>
          <td>2.0</td>
          <td>2.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>BEN_SEX_COD</th>
          <td>1.0</td>
          <td>2.00</td>
          <td>1.0</td>
          <td>2.0</td>
          <td>2.0</td>
        </tr>
        <tr>
          <th>DDP_SPE_COD</th>
          <td>0.0</td>
          <td>0.00</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>ETE_CAT_SNDS</th>
          <td>1101.0</td>
          <td>9999.00</td>
          <td>9999.0</td>
          <td>9999.0</td>
          <td>2206.0</td>
        </tr>
        <tr>
          <th>ETE_REG_COD</th>
          <td>76.0</td>
          <td>99.00</td>
          <td>99.0</td>
          <td>99.0</td>
          <td>32.0</td>
        </tr>
        <tr>
          <th>ETE_TYP_SNDS</th>
          <td>1.0</td>
          <td>99.00</td>
          <td>99.0</td>
          <td>99.0</td>
          <td>6.0</td>
        </tr>
        <tr>
          <th>ETP_REG_COD</th>
          <td>76.0</td>
          <td>24.00</td>
          <td>99.0</td>
          <td>99.0</td>
          <td>32.0</td>
        </tr>
        <tr>
          <th>ETP_CAT_SNDS</th>
          <td>1101.0</td>
          <td>1102.00</td>
          <td>9999.0</td>
          <td>9999.0</td>
          <td>2206.0</td>
        </tr>
        <tr>
          <th>MDT_TYP_COD</th>
          <td>9.0</td>
          <td>9.00</td>
          <td>9.0</td>
          <td>9.0</td>
          <td>9.0</td>
        </tr>
        <tr>
          <th>MFT_COD</th>
          <td>3.0</td>
          <td>99.00</td>
          <td>99.0</td>
          <td>99.0</td>
          <td>36.0</td>
        </tr>
        <tr>
          <th>PRS_FJH_TYP</th>
          <td>0.0</td>
          <td>0.00</td>
          <td>8.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>PRS_ACT_COG</th>
          <td>2.0</td>
          <td>1.00</td>
          <td>1.0</td>
          <td>2.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>PRS_ACT_NBR</th>
          <td>1.0</td>
          <td>5.00</td>
          <td>NaN</td>
          <td>2.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>PRS_ACT_QTE</th>
          <td>1.0</td>
          <td>1.00</td>
          <td>1.0</td>
          <td>2.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>PRS_DEP_MNT</th>
          <td>0.0</td>
          <td>0.00</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>PRS_PAI_MNT</th>
          <td>46.0</td>
          <td>0.51</td>
          <td>0.0</td>
          <td>46.0</td>
          <td>23.0</td>
        </tr>
        <tr>
          <th>PRS_REM_BSE</th>
          <td>46.0</td>
          <td>0.51</td>
          <td>0.0</td>
          <td>46.0</td>
          <td>23.0</td>
        </tr>
        <tr>
          <th>PRS_REM_MNT</th>
          <td>46.0</td>
          <td>0.51</td>
          <td>-0.5</td>
          <td>32.2</td>
          <td>23.0</td>
        </tr>
        <tr>
          <th>FLT_ACT_COG</th>
          <td>2.0</td>
          <td>1.00</td>
          <td>1.0</td>
          <td>2.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>FLT_ACT_NBR</th>
          <td>1.0</td>
          <td>5.00</td>
          <td>NaN</td>
          <td>2.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>FLT_ACT_QTE</th>
          <td>1.0</td>
          <td>1.00</td>
          <td>1.0</td>
          <td>2.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>FLT_PAI_MNT</th>
          <td>46.0</td>
          <td>0.51</td>
          <td>0.0</td>
          <td>46.0</td>
          <td>23.0</td>
        </tr>
        <tr>
          <th>FLT_DEP_MNT</th>
          <td>0.0</td>
          <td>0.00</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>FLT_REM_MNT</th>
          <td>46.0</td>
          <td>0.51</td>
          <td>-0.5</td>
          <td>32.2</td>
          <td>23.0</td>
        </tr>
        <tr>
          <th>SOI_ANN</th>
          <td>2016.0</td>
          <td>2016.00</td>
          <td>2016.0</td>
          <td>2016.0</td>
          <td>2016.0</td>
        </tr>
        <tr>
          <th>SOI_MOI</th>
          <td>12.0</td>
          <td>12.00</td>
          <td>12.0</td>
          <td>12.0</td>
          <td>12.0</td>
        </tr>
        <tr>
          <th>ASU_NAT</th>
          <td>40.0</td>
          <td>10.00</td>
          <td>10.0</td>
          <td>10.0</td>
          <td>10.0</td>
        </tr>
        <tr>
          <th>ATT_NAT</th>
          <td>0.0</td>
          <td>0.00</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>CPL_COD</th>
          <td>0.0</td>
          <td>0.00</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>CPT_ENV_TYP</th>
          <td>9.0</td>
          <td>9.00</td>
          <td>9.0</td>
          <td>9.0</td>
          <td>9.0</td>
        </tr>
        <tr>
          <th>DRG_AFF_NAT</th>
          <td>99.0</td>
          <td>99.00</td>
          <td>99.0</td>
          <td>99.0</td>
          <td>99.0</td>
        </tr>
        <tr>
          <th>ETE_IND_TAA</th>
          <td>1.0</td>
          <td>9.00</td>
          <td>9.0</td>
          <td>9.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>EXO_MTF</th>
          <td>99.0</td>
          <td>99.00</td>
          <td>99.0</td>
          <td>99.0</td>
          <td>99.0</td>
        </tr>
        <tr>
          <th>MTM_NAT</th>
          <td>9.0</td>
          <td>9.00</td>
          <td>9.0</td>
          <td>9.0</td>
          <td>9.0</td>
        </tr>
        <tr>
          <th>PRS_NAT</th>
          <td>1117.0</td>
          <td>3380.00</td>
          <td>1972.0</td>
          <td>1111.0</td>
          <td>1111.0</td>
        </tr>
        <tr>
          <th>PRS_PPU_SEC</th>
          <td>1.0</td>
          <td>2.00</td>
          <td>2.0</td>
          <td>2.0</td>
          <td>2.0</td>
        </tr>
        <tr>
          <th>PRS_REM_TAU</th>
          <td>100.0</td>
          <td>100.00</td>
          <td>100.0</td>
          <td>70.0</td>
          <td>100.0</td>
        </tr>
        <tr>
          <th>PRS_REM_TYP</th>
          <td>99.0</td>
          <td>99.00</td>
          <td>99.0</td>
          <td>99.0</td>
          <td>99.0</td>
        </tr>
        <tr>
          <th>PRS_PDS_QCP</th>
          <td>3.0</td>
          <td>31.00</td>
          <td>31.0</td>
          <td>10.0</td>
          <td>31.0</td>
        </tr>
        <tr>
          <th>EXE_INS_REG</th>
          <td>99.0</td>
          <td>28.00</td>
          <td>93.0</td>
          <td>32.0</td>
          <td>99.0</td>
        </tr>
        <tr>
          <th>PSE_ACT_SNDS</th>
          <td>0.0</td>
          <td>50.00</td>
          <td>50.0</td>
          <td>0.0</td>
          <td>19.0</td>
        </tr>
        <tr>
          <th>PSE_ACT_CAT</th>
          <td>0.0</td>
          <td>2.00</td>
          <td>2.0</td>
          <td>1.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>PSE_SPE_SNDS</th>
          <td>14.0</td>
          <td>0.00</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>PSE_STJ_SNDS</th>
          <td>2.0</td>
          <td>1.00</td>
          <td>9.0</td>
          <td>1.0</td>
          <td>2.0</td>
        </tr>
        <tr>
          <th>PRE_INS_REG</th>
          <td>99.0</td>
          <td>99.00</td>
          <td>93.0</td>
          <td>32.0</td>
          <td>99.0</td>
        </tr>
        <tr>
          <th>PSP_ACT_SNDS</th>
          <td>0.0</td>
          <td>0.00</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>PSP_ACT_CAT</th>
          <td>0.0</td>
          <td>0.00</td>
          <td>1.0</td>
          <td>1.0</td>
          <td>0.0</td>
        </tr>
        <tr>
          <th>PSP_SPE_SNDS</th>
          <td>1.0</td>
          <td>6.00</td>
          <td>1.0</td>
          <td>1.0</td>
          <td>1.0</td>
        </tr>
        <tr>
          <th>PSP_STJ_SNDS</th>
          <td>2.0</td>
          <td>2.00</td>
          <td>9.0</td>
          <td>1.0</td>
          <td>2.0</td>
        </tr>
        <tr>
          <th>TOP_PS5_TRG</th>
          <td>9.0</td>
          <td>9.00</td>
          <td>9.0</td>
          <td>9.0</td>
          <td>9.0</td>
        </tr>
        <tr>
          <th>Unnamed: 55</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    variables = pandas.read_excel(xlsfile, skiprows=2)
    cols = variables.columns[:2]
    variables[cols].dropna()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>AGE_BEN_SNDS</th>
          <th>Tranche d'Age Bénéficiaire au moment des soins</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>ASU_NAT</td>
          <td>Nature d'Assurance</td>
        </tr>
        <tr>
          <th>1</th>
          <td>ATT_NAT</td>
          <td>Nature de l'Accident du Travail</td>
        </tr>
        <tr>
          <th>2</th>
          <td>BEN_CMU_TOP</td>
          <td>Top Bénéficiaire CMU-C</td>
        </tr>
        <tr>
          <th>3</th>
          <td>BEN_QLT_COD</td>
          <td>Qualité du Bénéficiaire</td>
        </tr>
        <tr>
          <th>4</th>
          <td>BEN_RES_ZEAT</td>
          <td>ZEAT de Résidence du Bénéficiaire</td>
        </tr>
        <tr>
          <th>5</th>
          <td>BEN_SEX_COD</td>
          <td>Sexe du Bénéficiaire</td>
        </tr>
        <tr>
          <th>6</th>
          <td>CPL_COD</td>
          <td>Complément d'Acte</td>
        </tr>
        <tr>
          <th>7</th>
          <td>CPT_ENV_TYP</td>
          <td>Type d'Enveloppe</td>
        </tr>
        <tr>
          <th>8</th>
          <td>DDP_SPE_COD</td>
          <td>Discipline de Prestation Etb Exécutant</td>
        </tr>
        <tr>
          <th>9</th>
          <td>DRG_AFF_NAT</td>
          <td>Nature du Destinataire de Règlement affiné</td>
        </tr>
        <tr>
          <th>10</th>
          <td>ETE_CAT_SNDS</td>
          <td>Catégorie Etb Exécutant</td>
        </tr>
        <tr>
          <th>11</th>
          <td>ETE_IND_TAA</td>
          <td>Indicateur TAA Privé/Public</td>
        </tr>
        <tr>
          <th>12</th>
          <td>ETE_TYP_SNDS</td>
          <td>Type Etb Exécutant</td>
        </tr>
        <tr>
          <th>13</th>
          <td>ETE_ZEAT_COD</td>
          <td>ZEAT d'Implantation Etb Exécutant</td>
        </tr>
        <tr>
          <th>14</th>
          <td>ETP_CAT_SNDS</td>
          <td>Catégorie Etb Prescripteur</td>
        </tr>
        <tr>
          <th>15</th>
          <td>ETP_ZEAT_COD</td>
          <td>ZEAT d'Implantation Etb Prescripteur</td>
        </tr>
        <tr>
          <th>16</th>
          <td>EXE_INS_ZEAT</td>
          <td>ZEAT du PS Exécutant</td>
        </tr>
        <tr>
          <th>17</th>
          <td>EXO_MTF</td>
          <td>Motif d'Exonération du Ticket Modérateur</td>
        </tr>
        <tr>
          <th>18</th>
          <td>FLT_ACT_COG</td>
          <td>Coefficient Global de la Prestation Préfiltré</td>
        </tr>
        <tr>
          <th>19</th>
          <td>FLT_ACT_NBR</td>
          <td>Dénombrement de la Prestation Préfiltré</td>
        </tr>
        <tr>
          <th>20</th>
          <td>FLT_ACT_QTE</td>
          <td>Quantité de la Prestation Préfiltrée</td>
        </tr>
        <tr>
          <th>21</th>
          <td>FLT_DEP_MNT</td>
          <td>Montant du Dépassement de la Prestation Préfiltré</td>
        </tr>
        <tr>
          <th>22</th>
          <td>FLT_PAI_MNT</td>
          <td>Montant de la Dépense de la Prestation Préfiltrée</td>
        </tr>
        <tr>
          <th>23</th>
          <td>FLT_REM_MNT</td>
          <td>Montant Versé/Remboursé Préfiltré</td>
        </tr>
        <tr>
          <th>24</th>
          <td>FLX_ANN_MOI</td>
          <td>Année et Mois de Traitement</td>
        </tr>
        <tr>
          <th>25</th>
          <td>MDT_TYP_COD</td>
          <td>Mode de Traitement Etb Exécutant</td>
        </tr>
        <tr>
          <th>26</th>
          <td>MFT_COD</td>
          <td>Mode de Fixation des Tarifs Etb Exécutant</td>
        </tr>
        <tr>
          <th>27</th>
          <td>MTM_NAT</td>
          <td>Modulation du Ticket Modérateur</td>
        </tr>
        <tr>
          <th>28</th>
          <td>ORG_CLE_ZEAT</td>
          <td>ZEAT de l'Organisme de Liquidation</td>
        </tr>
        <tr>
          <th>29</th>
          <td>PRE_INS_ZEAT</td>
          <td>ZEAT du PS Prescripteur</td>
        </tr>
        <tr>
          <th>30</th>
          <td>PRS_ACT_COG</td>
          <td>Coefficient Global</td>
        </tr>
        <tr>
          <th>31</th>
          <td>PRS_ACT_NBR</td>
          <td>Dénombrement</td>
        </tr>
        <tr>
          <th>32</th>
          <td>PRS_ACT_QTE</td>
          <td>Quantité</td>
        </tr>
        <tr>
          <th>33</th>
          <td>PRS_DEP_MNT</td>
          <td>Montant du Dépassement</td>
        </tr>
        <tr>
          <th>34</th>
          <td>PRS_FJH_TYP</td>
          <td>Type de Prise en Charge Forfait Journalier</td>
        </tr>
        <tr>
          <th>35</th>
          <td>PRS_NAT</td>
          <td>Nature de Prestation</td>
        </tr>
        <tr>
          <th>36</th>
          <td>PRS_PAI_MNT</td>
          <td>Montant de la Dépense</td>
        </tr>
        <tr>
          <th>37</th>
          <td>PRS_PDS_QCP</td>
          <td>Code Qualificatif Parcours de Soins (sortie)</td>
        </tr>
        <tr>
          <th>38</th>
          <td>PRS_PPU_SEC</td>
          <td>Code Secteur Privé/Public</td>
        </tr>
        <tr>
          <th>39</th>
          <td>PRS_REM_BSE</td>
          <td>Base de Remboursement</td>
        </tr>
        <tr>
          <th>40</th>
          <td>PRS_REM_MNT</td>
          <td>Montant Versé/Remboursé</td>
        </tr>
        <tr>
          <th>41</th>
          <td>PRS_REM_TAU</td>
          <td>Taux de Remboursement</td>
        </tr>
        <tr>
          <th>42</th>
          <td>PRS_REM_TYP</td>
          <td>Type de Remboursement</td>
        </tr>
        <tr>
          <th>43</th>
          <td>PSE_ACT_CAT</td>
          <td>Catégorie de l' Exécutant</td>
        </tr>
        <tr>
          <th>44</th>
          <td>PSE_ACT_SNDS</td>
          <td>Nature d'Activité PS Exécutant</td>
        </tr>
        <tr>
          <th>45</th>
          <td>PSE_SPE_SNDS</td>
          <td>Spécialité Médicale PS Exécutant</td>
        </tr>
        <tr>
          <th>46</th>
          <td>PSE_STJ_SNDS</td>
          <td>Statut Juridique PS Exécutant</td>
        </tr>
        <tr>
          <th>47</th>
          <td>PSP_ACT_CAT</td>
          <td>Catégorie du Prescripteur</td>
        </tr>
        <tr>
          <th>48</th>
          <td>PSP_ACT_SNDS</td>
          <td>Nature d'Activité PS Prescripteur</td>
        </tr>
        <tr>
          <th>49</th>
          <td>PSP_SPE_SNDS</td>
          <td>Spécialité Médicale PS Prescripteur</td>
        </tr>
        <tr>
          <th>50</th>
          <td>PSP_STJ_SNDS</td>
          <td>Statut Juridique PS Prescripteur</td>
        </tr>
        <tr>
          <th>51</th>
          <td>SOI_ANN</td>
          <td>Année de Soins</td>
        </tr>
        <tr>
          <th>52</th>
          <td>SOI_MOI</td>
          <td>Mois de Soins</td>
        </tr>
        <tr>
          <th>53</th>
          <td>TOP_PS5_TRG</td>
          <td>Top Périmètre hors CMU C et prestations pour i...</td>
        </tr>
      </tbody>
    </table>
    </div>



Répartition des dépenses selon l’âge
------------------------------------

C’est la variable ``AGE_BEN_SNDS`` mais on ne peut toujours pas charger
le fichier en mémoire.

Itérateur sur des dataframes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Il faut utiliser le paramètre
`iterator <http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html>`__.

.. code:: ipython3

    dfs = []
    for i, df in enumerate(pandas.read_csv(filename, 
                sep=";", iterator=True, chunksize=10000)):
        dfs.append(df)
        if i >= 5:
            break        
    dfs[0].head()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>FLX_ANN_MOI</th>
          <th>ORG_CLE_REG</th>
          <th>AGE_BEN_SNDS</th>
          <th>BEN_RES_REG</th>
          <th>BEN_CMU_TOP</th>
          <th>BEN_QLT_COD</th>
          <th>BEN_SEX_COD</th>
          <th>DDP_SPE_COD</th>
          <th>ETE_CAT_SNDS</th>
          <th>ETE_REG_COD</th>
          <th>...</th>
          <th>PSE_ACT_CAT</th>
          <th>PSE_SPE_SNDS</th>
          <th>PSE_STJ_SNDS</th>
          <th>PRE_INS_REG</th>
          <th>PSP_ACT_SNDS</th>
          <th>PSP_ACT_CAT</th>
          <th>PSP_SPE_SNDS</th>
          <th>PSP_STJ_SNDS</th>
          <th>TOP_PS5_TRG</th>
          <th>Unnamed: 55</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>201612</td>
          <td>76</td>
          <td>20</td>
          <td>76</td>
          <td>9</td>
          <td>1</td>
          <td>1</td>
          <td>0</td>
          <td>1101</td>
          <td>76</td>
          <td>...</td>
          <td>0</td>
          <td>14</td>
          <td>2</td>
          <td>99</td>
          <td>0</td>
          <td>0</td>
          <td>1</td>
          <td>2</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>1</th>
          <td>201612</td>
          <td>28</td>
          <td>30</td>
          <td>28</td>
          <td>9</td>
          <td>1</td>
          <td>2</td>
          <td>0</td>
          <td>9999</td>
          <td>99</td>
          <td>...</td>
          <td>2</td>
          <td>0</td>
          <td>1</td>
          <td>99</td>
          <td>0</td>
          <td>0</td>
          <td>6</td>
          <td>2</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>2</th>
          <td>201612</td>
          <td>27</td>
          <td>60</td>
          <td>93</td>
          <td>9</td>
          <td>2</td>
          <td>1</td>
          <td>0</td>
          <td>9999</td>
          <td>99</td>
          <td>...</td>
          <td>2</td>
          <td>0</td>
          <td>9</td>
          <td>93</td>
          <td>0</td>
          <td>1</td>
          <td>1</td>
          <td>9</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>3</th>
          <td>201612</td>
          <td>32</td>
          <td>40</td>
          <td>32</td>
          <td>9</td>
          <td>2</td>
          <td>2</td>
          <td>0</td>
          <td>9999</td>
          <td>99</td>
          <td>...</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>32</td>
          <td>0</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>4</th>
          <td>201612</td>
          <td>32</td>
          <td>30</td>
          <td>32</td>
          <td>9</td>
          <td>1</td>
          <td>2</td>
          <td>0</td>
          <td>2206</td>
          <td>32</td>
          <td>...</td>
          <td>0</td>
          <td>0</td>
          <td>2</td>
          <td>99</td>
          <td>0</td>
          <td>0</td>
          <td>1</td>
          <td>2</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
      </tbody>
    </table>
    <p>5 rows × 56 columns</p>
    </div>



Il faut calculer la distribution des âges pour chaque dataframe.

.. code:: ipython3

    from datetime import datetime
    ages = []
    for i, df in enumerate(pandas.read_csv(filename, 
                        sep=";", iterator=True, chunksize=10000)):
        sub = df[["AGE_BEN_SNDS", "PRS_REM_MNT"]].copy()
        gr = sub.groupby("AGE_BEN_SNDS").sum()
        ages.append(gr)
        if len(ages) % 100 == 0:
            print(datetime.now(), len(ages), "lignes", 10000 * len(ages))

Les versions de pandas ont quelques différences. Voici celle utilisées
pour ce notebook.

.. code:: ipython3

    pandas.__version__




.. parsed-literal::
    '1.1.0'



.. code:: ipython3

    age = pandas.concat(ages)
    agg = age.groupby("AGE_BEN_SNDS").sum()
    agg.plot(kind="bar");



.. image:: bigdata_ways_22_0.png


Difficile d’interpréter ce graphique sans le comparer à la pyramide des
âges mais ce n’est pas l’objectif ici. Chaque statistique prend un peu
de temps. Et si nous tirions un échantillon aléatoire à 1%.

Echantillon
-----------

.. code:: ipython3

    ech = []
    for i, df in enumerate(pandas.read_csv(filename, 
                        sep=";", iterator=True, chunksize=10000)):
        sub = df.sample(frac=0.01)
        ech.append(sub)
        if len(ech) % 100 == 0:
            print(datetime.now(), len(ech), "lignes", 10000 * len(ech))

.. code:: ipython3

    echantillon = pandas.concat(ech)
    echantillon.shape




.. parsed-literal::
    (1000, 56)



.. code:: ipython3

    echantillon[["AGE_BEN_SNDS", "PRS_REM_MNT"]] \
            .groupby("AGE_BEN_SNDS").sum().plot(kind="bar");



.. image:: bigdata_ways_27_0.png


pickle
------

Le module `pickle <https://docs.python.org/3/library/pickle.html>`__
permet de
`sérialiser <https://fr.wikipedia.org/wiki/S%C3%A9rialisation>`__. Cette
opération transforme le contenu d’un objet tel qu’il est en mémoire en
une séquence d’octets qu’on peut alors enregistrer sur disque. Comme ce
format de fichier est plus proche de la mémoire, il se charge beaucoup
plus vite. Cela permet de gagner un peu de temps. L’inconvénient est que
ce format n’est pas toujours compatible d’une version de python ou
pandas à l’autre.

.. code:: ipython3

    echantillon.to_csv("echantillon.csv", sep=";", index=False)

.. code:: ipython3

    import os
    os.stat("echantillon.csv").st_size / 2**20




.. parsed-literal::
    0.16968822479248047



.. code:: ipython3

    begin = time.perf_counter()
    ech2 = pandas.read_csv("echantillon.csv", sep=";")
    print(time.perf_counter()-begin)


.. parsed-literal::
    0.025167299999992565


.. code:: ipython3

    ech2.shape




.. parsed-literal::
    (1000, 56)



.. code:: ipython3

    echantillon.to_pickle("echantillon.pickle")

.. code:: ipython3

    begin = time.perf_counter()
    ech3 = pandas.read_pickle("echantillon.pickle")
    print(time.perf_counter()-begin)


.. parsed-literal::
    0.011920500000002221


SQLite3
~~~~~~~

Un échantillon fait disparaître les classes les moins représentées. On
peut accélérer le traitement en passant par
`Sqlite <https://www.sqlite.org/>`__ et disponible via le module
`sqlite <https://docs.python.org/3.6/library/sqlite.html>`__. Première
étape : créer la base.

.. code:: ipython3

    import sqlite3
    cnx = sqlite3.connect('damir2.db3')

.. code:: ipython3

    for i, df in enumerate(pandas.read_csv(filename, 
                sep=";", iterator=True, chunksize=100000)):
        df.to_sql(con=cnx, name="damir", if_exists="replace" if i == 0 else "append")
        if (i+1) % 10 == 0:
            print(datetime.now(), (i+1) * 100000, "lignes")
    print("done")


.. parsed-literal::
    c:\python372_x64\lib\site-packages\pandas\core\generic.py:2612: UserWarning: The spaces in these column names will not be changed. In pandas versions < 0.14, spaces were converted to underscores.
      method=method,


.. parsed-literal::

    done


C’est un peu long mais le calcul de statistique est plus rapide par la
suite.

.. code:: ipython3

    import time
    begin = time.perf_counter()
    dfage = pandas.read_sql(
        'select AGE_BEN_SNDS, count(*) from damir group by AGE_BEN_SNDS', cnx)
    print("time", time.perf_counter() - begin)
    dfage


.. parsed-literal::
    time 0.08782719999999244






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>AGE_BEN_SNDS</th>
          <th>count(*)</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0</td>
          <td>10424</td>
        </tr>
        <tr>
          <th>1</th>
          <td>20</td>
          <td>10948</td>
        </tr>
        <tr>
          <th>2</th>
          <td>30</td>
          <td>10871</td>
        </tr>
        <tr>
          <th>3</th>
          <td>40</td>
          <td>11963</td>
        </tr>
        <tr>
          <th>4</th>
          <td>50</td>
          <td>14475</td>
        </tr>
        <tr>
          <th>5</th>
          <td>60</td>
          <td>15662</td>
        </tr>
        <tr>
          <th>6</th>
          <td>70</td>
          <td>13220</td>
        </tr>
        <tr>
          <th>7</th>
          <td>80</td>
          <td>12281</td>
        </tr>
        <tr>
          <th>8</th>
          <td>99</td>
          <td>155</td>
        </tr>
      </tbody>
    </table>
    </div>



On peut aussi programmer des `commandes
magiques <http://www.xavierdupre.fr/app/pyensae/helpsphinx/notebooks/pyensae_sql_magic.html>`__
pour alléger la syntaxe.

.. code:: ipython3

    %load_ext pyensae

.. code:: ipython3

    %SQL_connect damir.db3




.. parsed-literal::
    <pyensae.sql.sql_interface_database.InterfaceSQLDatabase at 0x1dc5c25add8>



.. code:: ipython3

    %SQL_tables




.. parsed-literal::
    []



.. code:: ipython3

    %%SQL
    SELECT * FROM damir LIMIT 5


.. parsed-literal::
    unable to execute a SQL request (1)(file damir.db3)
    SELECT * FROM damir LIMIT 5
    no such table: damir
    'SELECT * FROM damir LIMIT 5'


.. code:: ipython3

    %lsmagic




.. parsed-literal::
    Available line magics:
    %SQL  %SQL_add_function  %SQL_close  %SQL_connect  %SQL_drop_table  %SQL_import_df  %SQL_import_tsv  %SQL_refresh_completion  %SQL_schema  %SQL_tables  %alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cd  %clear  %cls  %colors  %conda  %config  %connect_info  %copy  %ddir  %debug  %dhist  %dirs  %doctest_mode  %echo  %ed  %edit  %encoding  %env  %grep  %gui  %head  %hhelp  %hist  %history  %jsdf  %killbgscripts  %ldir  %less  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lsr  %lsrepo  %macro  %magic  %matplotlib  %mkdir  %more  %mpl_style  %nb_menu  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %ren  %rep  %rerun  %reset  %reset_selective  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tail  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode
    Available cell magics:
    %%!  %%HTML  %%PYTHON  %%SQL  %%SVG  %%bash  %%capture  %%cmd  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%runpy  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile
    Automagic is ON, % prefix IS NOT needed for line magics.



dask
~~~~

`dask <http://dask.pydata.org/en/latest/>`__ fonctionne un peu comme
pandas excepté que l’exécution des requêtes est différée. On utilise
`read_csv <http://dask.pydata.org/en/latest/dataframe-api.html#dask.dataframe.read_csv>`__.

.. code:: ipython3

    import dask.dataframe
    dfdk = dask.dataframe.read_csv(filename, sep=";")
    dfdk.head(n=2)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>FLX_ANN_MOI</th>
          <th>ORG_CLE_REG</th>
          <th>AGE_BEN_SNDS</th>
          <th>BEN_RES_REG</th>
          <th>BEN_CMU_TOP</th>
          <th>BEN_QLT_COD</th>
          <th>BEN_SEX_COD</th>
          <th>DDP_SPE_COD</th>
          <th>ETE_CAT_SNDS</th>
          <th>ETE_REG_COD</th>
          <th>...</th>
          <th>PSE_ACT_CAT</th>
          <th>PSE_SPE_SNDS</th>
          <th>PSE_STJ_SNDS</th>
          <th>PRE_INS_REG</th>
          <th>PSP_ACT_SNDS</th>
          <th>PSP_ACT_CAT</th>
          <th>PSP_SPE_SNDS</th>
          <th>PSP_STJ_SNDS</th>
          <th>TOP_PS5_TRG</th>
          <th>Unnamed: 55</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>201612</td>
          <td>76</td>
          <td>20</td>
          <td>76</td>
          <td>9</td>
          <td>1</td>
          <td>1</td>
          <td>0</td>
          <td>1101</td>
          <td>76</td>
          <td>...</td>
          <td>0</td>
          <td>14</td>
          <td>2</td>
          <td>99</td>
          <td>0</td>
          <td>0</td>
          <td>1</td>
          <td>2</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>1</th>
          <td>201612</td>
          <td>28</td>
          <td>30</td>
          <td>28</td>
          <td>9</td>
          <td>1</td>
          <td>2</td>
          <td>0</td>
          <td>9999</td>
          <td>99</td>
          <td>...</td>
          <td>2</td>
          <td>0</td>
          <td>1</td>
          <td>99</td>
          <td>0</td>
          <td>0</td>
          <td>6</td>
          <td>2</td>
          <td>9</td>
          <td>NaN</td>
        </tr>
      </tbody>
    </table>
    <p>2 rows × 56 columns</p>
    </div>



Les trois lignes qui suivent sont très rapides.

.. code:: ipython3

    import dask
    dask.__version__




.. parsed-literal::
    '2.12.0'



.. code:: ipython3

    dfdk2 = dfdk[["AGE_BEN_SNDS", "PRS_REM_MNT"]]
    grdk = dfdk2.groupby("AGE_BEN_SNDS").sum()

La suivante, beaucoup moins.

.. code:: ipython3

    begin = time.perf_counter()
    res = grdk.compute()
    print(time.perf_counter() - begin)


.. parsed-literal::
    0.5088113999999848


.. code:: ipython3

    ech4 = dfdk.sample(frac=0.01)

.. code:: ipython3

    dfech = ech4.compute()

.. code:: ipython3

    type(dfech)




.. parsed-literal::
    pandas.core.frame.DataFrame



.. code:: ipython3

    res






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>PRS_REM_MNT</th>
        </tr>
        <tr>
          <th>AGE_BEN_SNDS</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>2607345.16</td>
        </tr>
        <tr>
          <th>20</th>
          <td>1714894.46</td>
        </tr>
        <tr>
          <th>30</th>
          <td>3378001.46</td>
        </tr>
        <tr>
          <th>40</th>
          <td>3271108.13</td>
        </tr>
        <tr>
          <th>50</th>
          <td>5011475.59</td>
        </tr>
        <tr>
          <th>60</th>
          <td>6026792.72</td>
        </tr>
        <tr>
          <th>70</th>
          <td>4283277.72</td>
        </tr>
        <tr>
          <th>80</th>
          <td>4181565.36</td>
        </tr>
        <tr>
          <th>99</th>
          <td>19015.33</td>
        </tr>
      </tbody>
    </table>
    </div>



C’est plus qu’avec SQLite3 mais dans cet exemple, dask relit le fichier
à chaque calcul car il ne stocke pas un dataframe dans un format
intermédiaire plus efficace tel que
`parquet <http://parquet.apache.org/>`__. dask propose des idées pour
optimiser les calculs `Dask DataFrame Performance
Tips <http://dask.pydata.org/en/latest/dataframe-performance.html>`__.

Grandes matrices
----------------

Les dataframes acceptent un type différent pour chaque colonne. Si
toutes les colonnes sont numériques et de même types, on peut s’orienter
vers d’autres solutions.

Zarr
~~~~

`zarr <http://zarr.readthedocs.io/en/latest/>`__ permet de manipuler des
matrices qui ne tiennent pas en mémoire de la même façon que
`numpy <http://www.numpy.org/>`__. Les données sont compressés et
parfois stockées sur disque. Ce module masque beaucoup de détails en
proposant une interface connue.

.. code:: ipython3

    import zarr
    z = zarr.zeros((10000, 10000), chunks=(1000, 1000), dtype='i4')
    z




.. parsed-literal::
    <zarr.core.Array (10000, 10000) int32>



.. code:: ipython3

    z[4,5]




.. parsed-literal::
    0



Voir aussi
`wendelin.core <https://pypi.python.org/pypi/wendelin.core>`__,
`hdf5 <http://www.h5py.org/>`__.

Formats compressés
------------------


-  `sparse
   dataframes <https://pandas.pydata.org/pandas-docs/stable/sparse.html#>`__
-  `scipy <https://docs.scipy.org/doc/scipy-0.18.1/reference/sparse.html>`__