.. _2020jsonxmlrst:

=================
Tech - JSON - XML
=================


.. only:: html

    **Links:** :download:`notebook <2020_json_xml.ipynb>`, :downloadlink:`html <2020_json_xml2html.html>`, :download:`python <2020_json_xml.py>`, :downloadlink:`slides <2020_json_xml.slides.html>`, :githublink:`GitHub|_doc/notebooks/td1a_home/2020_json_xml.ipynb|*`


Transmettre l’information d’une machine à une autre, d’un logiciel à un
autre, d’une base de données à une autre est un problème récurrent. Le
format le plus simple pour des données est le format
`csv <https://fr.wikipedia.org/wiki/Comma-separated_values>`__. Ca
marche bien pour les tables mais cela ne permet de transmettre aisément
des `données non
structurées <https://fr.wikipedia.org/wiki/Informations_non_structur%C3%A9es>`__.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





Enoncé
------

.. code:: ipython3

    from sklearn.datasets import load_iris as load_data
    from pandas import DataFrame
    data = load_data()
    df = DataFrame(data.data, columns=data.feature_names)
    df['fleur'] = [data.target_names[t] for t in data.target]
    df.tail()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>sepal length (cm)</th>
          <th>sepal width (cm)</th>
          <th>petal length (cm)</th>
          <th>petal width (cm)</th>
          <th>fleur</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>145</th>
          <td>6.7</td>
          <td>3.0</td>
          <td>5.2</td>
          <td>2.3</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>146</th>
          <td>6.3</td>
          <td>2.5</td>
          <td>5.0</td>
          <td>1.9</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>147</th>
          <td>6.5</td>
          <td>3.0</td>
          <td>5.2</td>
          <td>2.0</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>148</th>
          <td>6.2</td>
          <td>3.4</td>
          <td>5.4</td>
          <td>2.3</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>149</th>
          <td>5.9</td>
          <td>3.0</td>
          <td>5.1</td>
          <td>1.8</td>
          <td>virginica</td>
        </tr>
      </tbody>
    </table>
    </div>



Q1 : écriture des données au format CSV
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Le plus simple…

.. code:: ipython3

    from io import StringIO
    buffer = StringIO()
    df.to_csv(buffer, index=False)
    text = buffer.getvalue()
    text[:300]




.. parsed-literal::
    'sepal length (cm),sepal width (cm),petal length (cm),petal width (cm),fleur\r\n5.1,3.5,1.4,0.2,setosa\r\n4.9,3.0,1.4,0.2,setosa\r\n4.7,3.2,1.3,0.2,setosa\r\n4.6,3.1,1.5,0.2,setosa\r\n5.0,3.6,1.4,0.2,setosa\r\n5.4,3.9,1.7,0.4,setosa\r\n4.6,3.4,1.4,0.3,setosa\r\n5.0,3.4,1.5,0.2,setosa\r\n4.4,2.9,1.4,0.2,setosa\r\n4.9,3.1'




Q2 : écriture des données au format JSON
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: ipython3

    r = df.to_json(orient='records')
    r[:400]




.. parsed-literal::
    '[{"sepal length (cm)":5.1,"sepal width (cm)":3.5,"petal length (cm)":1.4,"petal width (cm)":0.2,"fleur":"setosa"},{"sepal length (cm)":4.9,"sepal width (cm)":3.0,"petal length (cm)":1.4,"petal width (cm)":0.2,"fleur":"setosa"},{"sepal length (cm)":4.7,"sepal width (cm)":3.2,"petal length (cm)":1.3,"petal width (cm)":0.2,"fleur":"setosa"},{"sepal length (cm)":4.6,"sepal width (cm)":3.1,"petal lengt'




Q3 : relire les données avec le module `json <https://docs.python.org/3/library/json.html>`__
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



Q4 : essayez avec les format XML (ou HTML), SQL, SAS, Excel…
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Q5 : données non structurées
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On ajoute les endroits où ces fleurs sont présentes. On voudrait que
toutes les informations soient présentes dans le même fichier. Comment
fait-on ?

.. code:: ipython3

    locations = {'virginica': ['Florida', 'Georgia'],
                 'setosa': ['Maine', 'Alaska', 'Quebec'],
                 'versicolor': ['Quebec', 'Georgia', 'Ireland', 'Main']}

La question sous-jacente est : vaut-il mieux avoir deux fichiers plats,
l’un pour les données décrivant les fleurs, l’autre pour les
localisations ou un seul fusionnant les deux informations ?


Q6 : le texte, ça prend trop de place, zippons
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Avec le module
`zipfile <https://docs.python.org/3/library/zipfile.html>`__.


Q7 : que vous inspire `protobuf <https://developers.google.com/protocol-buffers>`__ ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Ou `flatbuffers <https://google.github.io/flatbuffers/>`__,
`MessagePack <https://msgpack.org/>`__


Réponses
--------

Q1 : écriture des données au format CSV
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: ipython3

    from io import StringIO
    buffer = StringIO()
    df.to_csv(buffer, index=False)
    text = buffer.getvalue()
    text[:300]




.. parsed-literal::
    'sepal length (cm),sepal width (cm),petal length (cm),petal width (cm),fleur\r\n5.1,3.5,1.4,0.2,setosa\r\n4.9,3.0,1.4,0.2,setosa\r\n4.7,3.2,1.3,0.2,setosa\r\n4.6,3.1,1.5,0.2,setosa\r\n5.0,3.6,1.4,0.2,setosa\r\n5.4,3.9,1.7,0.4,setosa\r\n4.6,3.4,1.4,0.3,setosa\r\n5.0,3.4,1.5,0.2,setosa\r\n4.4,2.9,1.4,0.2,setosa\r\n4.9,3.1'



.. code:: ipython3

    df.to_csv("fleurs.csv", index=False)

.. code:: ipython3

    import os
    os.listdir(".")




.. parsed-literal::
    ['.ipynb_checkpoints',
     '2020_covid.ipynb',
     '2020_edit.ipynb',
     '2020_json_xml.ipynb',
     '2020_numpy.ipynb',
     '2020_pandas.ipynb',
     '2020_profile.ipynb',
     '2020_regex.ipynb',
     '2020_suffix.ipynb',
     '2020_surface.ipynb',
     '2020_topk.ipynb',
     '2020_tsp.ipynb',
     'data.csv',
     'fleurs.csv']



.. code:: ipython3

    import pandas
    df2 = pandas.read_csv("fleurs.csv")

.. code:: ipython3

    df2.head()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>sepal length (cm)</th>
          <th>sepal width (cm)</th>
          <th>petal length (cm)</th>
          <th>petal width (cm)</th>
          <th>fleur</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4.9</td>
          <td>3.0</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>2</th>
          <td>4.7</td>
          <td>3.2</td>
          <td>1.3</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>3</th>
          <td>4.6</td>
          <td>3.1</td>
          <td>1.5</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>4</th>
          <td>5.0</td>
          <td>3.6</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    virtuel = StringIO(text)
    df3 = pandas.read_csv(virtuel)
    df3.head()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>sepal length (cm)</th>
          <th>sepal width (cm)</th>
          <th>petal length (cm)</th>
          <th>petal width (cm)</th>
          <th>fleur</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4.9</td>
          <td>3.0</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>2</th>
          <td>4.7</td>
          <td>3.2</td>
          <td>1.3</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>3</th>
          <td>4.6</td>
          <td>3.1</td>
          <td>1.5</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>4</th>
          <td>5.0</td>
          <td>3.6</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
      </tbody>
    </table>
    </div>



Q2 : écriture des données au format JSON
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: ipython3

    json_text = df.to_json(orient='records')
    json_text[:400]




.. parsed-literal::
    '[{"sepal length (cm)":5.1,"sepal width (cm)":3.5,"petal length (cm)":1.4,"petal width (cm)":0.2,"fleur":"setosa"},{"sepal length (cm)":4.9,"sepal width (cm)":3.0,"petal length (cm)":1.4,"petal width (cm)":0.2,"fleur":"setosa"},{"sepal length (cm)":4.7,"sepal width (cm)":3.2,"petal length (cm)":1.3,"petal width (cm)":0.2,"fleur":"setosa"},{"sepal length (cm)":4.6,"sepal width (cm)":3.1,"petal lengt'



Q3 : relire les données avec le module `json <https://docs.python.org/3/library/json.html>`__
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: ipython3

    import json

.. code:: ipython3

    res = json.loads(json_text)

.. code:: ipython3

    for i, r in enumerate(res):
        print(i, type(r), r)
        if i >= 5:
            break


.. parsed-literal::
    0 <class 'dict'> {'sepal length (cm)': 5.1, 'sepal width (cm)': 3.5, 'petal length (cm)': 1.4, 'petal width (cm)': 0.2, 'fleur': 'setosa'}
    1 <class 'dict'> {'sepal length (cm)': 4.9, 'sepal width (cm)': 3.0, 'petal length (cm)': 1.4, 'petal width (cm)': 0.2, 'fleur': 'setosa'}
    2 <class 'dict'> {'sepal length (cm)': 4.7, 'sepal width (cm)': 3.2, 'petal length (cm)': 1.3, 'petal width (cm)': 0.2, 'fleur': 'setosa'}
    3 <class 'dict'> {'sepal length (cm)': 4.6, 'sepal width (cm)': 3.1, 'petal length (cm)': 1.5, 'petal width (cm)': 0.2, 'fleur': 'setosa'}
    4 <class 'dict'> {'sepal length (cm)': 5.0, 'sepal width (cm)': 3.6, 'petal length (cm)': 1.4, 'petal width (cm)': 0.2, 'fleur': 'setosa'}
    5 <class 'dict'> {'sepal length (cm)': 5.4, 'sepal width (cm)': 3.9, 'petal length (cm)': 1.7, 'petal width (cm)': 0.4, 'fleur': 'setosa'}


.. code:: ipython3

    res[3]['sepal width (cm)']




.. parsed-literal::
    3.1



.. code:: ipython3

    virtuel = StringIO(json_text)
    res2 = json.load(virtuel)
    res2[:3]




.. parsed-literal::
    [{'sepal length (cm)': 5.1,
      'sepal width (cm)': 3.5,
      'petal length (cm)': 1.4,
      'petal width (cm)': 0.2,
      'fleur': 'setosa'},
     {'sepal length (cm)': 4.9,
      'sepal width (cm)': 3.0,
      'petal length (cm)': 1.4,
      'petal width (cm)': 0.2,
      'fleur': 'setosa'},
     {'sepal length (cm)': 4.7,
      'sepal width (cm)': 3.2,
      'petal length (cm)': 1.3,
      'petal width (cm)': 0.2,
      'fleur': 'setosa'}]



Q4 : essayez avec les format XML (ou HTML), SQL, SAS, Excel…
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: ipython3

    html_text = df.to_html(index=False)

.. code:: ipython3

    print(html_text[:500])


.. parsed-literal::
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th>sepal length (cm)</th>
          <th>sepal width (cm)</th>
          <th>petal length (cm)</th>
          <th>petal width (cm)</th>
          <th>fleur</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td>5.1</td>
          <td>3.5</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <td>4.9</td>
          <td>3.0</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
     


.. code:: ipython3

    df_html = pandas.read_html(html_text)
    df_html[0].tail()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>sepal length (cm)</th>
          <th>sepal width (cm)</th>
          <th>petal length (cm)</th>
          <th>petal width (cm)</th>
          <th>fleur</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>145</th>
          <td>6.7</td>
          <td>3.0</td>
          <td>5.2</td>
          <td>2.3</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>146</th>
          <td>6.3</td>
          <td>2.5</td>
          <td>5.0</td>
          <td>1.9</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>147</th>
          <td>6.5</td>
          <td>3.0</td>
          <td>5.2</td>
          <td>2.0</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>148</th>
          <td>6.2</td>
          <td>3.4</td>
          <td>5.4</td>
          <td>2.3</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>149</th>
          <td>5.9</td>
          <td>3.0</td>
          <td>5.1</td>
          <td>1.8</td>
          <td>virginica</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    df_html = pandas.read_html(html_text + html_text)
    len(df_html)




.. parsed-literal::
    2



Q5 : données non structurées
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: ipython3

    df.head()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>sepal length (cm)</th>
          <th>sepal width (cm)</th>
          <th>petal length (cm)</th>
          <th>petal width (cm)</th>
          <th>fleur</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4.9</td>
          <td>3.0</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>2</th>
          <td>4.7</td>
          <td>3.2</td>
          <td>1.3</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>3</th>
          <td>4.6</td>
          <td>3.1</td>
          <td>1.5</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
        <tr>
          <th>4</th>
          <td>5.0</td>
          <td>3.6</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    locations = {'virginica': ['Florida', 'Georgia'],
                 'setosa': ['Maine', 'Alaska', 'Quebec'],
                 'versicolor': ['Quebec', 'Georgia', 'Ireland', 'Main']}

.. code:: ipython3

    obs = []
    for fleur, loc in locations.items():
        for l in loc:
            obs.append({"fleur": fleur, "location": l})
    obs




.. parsed-literal::
    [{'fleur': 'virginica', 'location': 'Florida'},
     {'fleur': 'virginica', 'location': 'Georgia'},
     {'fleur': 'setosa', 'location': 'Maine'},
     {'fleur': 'setosa', 'location': 'Alaska'},
     {'fleur': 'setosa', 'location': 'Quebec'},
     {'fleur': 'versicolor', 'location': 'Quebec'},
     {'fleur': 'versicolor', 'location': 'Georgia'},
     {'fleur': 'versicolor', 'location': 'Ireland'},
     {'fleur': 'versicolor', 'location': 'Main'}]



.. code:: ipython3

    df_locations = pandas.DataFrame(obs)
    df_locations






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>fleur</th>
          <th>location</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>virginica</td>
          <td>Florida</td>
        </tr>
        <tr>
          <th>1</th>
          <td>virginica</td>
          <td>Georgia</td>
        </tr>
        <tr>
          <th>2</th>
          <td>setosa</td>
          <td>Maine</td>
        </tr>
        <tr>
          <th>3</th>
          <td>setosa</td>
          <td>Alaska</td>
        </tr>
        <tr>
          <th>4</th>
          <td>setosa</td>
          <td>Quebec</td>
        </tr>
        <tr>
          <th>5</th>
          <td>versicolor</td>
          <td>Quebec</td>
        </tr>
        <tr>
          <th>6</th>
          <td>versicolor</td>
          <td>Georgia</td>
        </tr>
        <tr>
          <th>7</th>
          <td>versicolor</td>
          <td>Ireland</td>
        </tr>
        <tr>
          <th>8</th>
          <td>versicolor</td>
          <td>Main</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    merged = df.merge(df_locations, left_on="fleur", right_on="fleur")
    merged.head(10)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>sepal length (cm)</th>
          <th>sepal width (cm)</th>
          <th>petal length (cm)</th>
          <th>petal width (cm)</th>
          <th>fleur</th>
          <th>location</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>Maine</td>
        </tr>
        <tr>
          <th>1</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>Alaska</td>
        </tr>
        <tr>
          <th>2</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>Quebec</td>
        </tr>
        <tr>
          <th>3</th>
          <td>4.9</td>
          <td>3.0</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>Maine</td>
        </tr>
        <tr>
          <th>4</th>
          <td>4.9</td>
          <td>3.0</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>Alaska</td>
        </tr>
        <tr>
          <th>5</th>
          <td>4.9</td>
          <td>3.0</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>Quebec</td>
        </tr>
        <tr>
          <th>6</th>
          <td>4.7</td>
          <td>3.2</td>
          <td>1.3</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>Maine</td>
        </tr>
        <tr>
          <th>7</th>
          <td>4.7</td>
          <td>3.2</td>
          <td>1.3</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>Alaska</td>
        </tr>
        <tr>
          <th>8</th>
          <td>4.7</td>
          <td>3.2</td>
          <td>1.3</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>Quebec</td>
        </tr>
        <tr>
          <th>9</th>
          <td>4.6</td>
          <td>3.1</td>
          <td>1.5</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>Maine</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    merged.shape




.. parsed-literal::
    (450, 6)



.. code:: ipython3

    locations




.. parsed-literal::
    {'virginica': ['Florida', 'Georgia'],
     'setosa': ['Maine', 'Alaska', 'Quebec'],
     'versicolor': ['Quebec', 'Georgia', 'Ireland', 'Main']}



.. code:: ipython3

    obs2 = []
    for fleur, loc in locations.items():
        obs2.append({"fleur": fleur, "location": loc})
    obs2




.. parsed-literal::
    [{'fleur': 'virginica', 'location': ['Florida', 'Georgia']},
     {'fleur': 'setosa', 'location': ['Maine', 'Alaska', 'Quebec']},
     {'fleur': 'versicolor', 'location': ['Quebec', 'Georgia', 'Ireland', 'Main']}]



.. code:: ipython3

    df_locations2 = pandas.DataFrame(obs2)
    df_locations2






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>fleur</th>
          <th>location</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>virginica</td>
          <td>[Florida, Georgia]</td>
        </tr>
        <tr>
          <th>1</th>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
        <tr>
          <th>2</th>
          <td>versicolor</td>
          <td>[Quebec, Georgia, Ireland, Main]</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    merged = df.merge(df_locations2, left_on="fleur", right_on="fleur")
    merged.head(10)






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>sepal length (cm)</th>
          <th>sepal width (cm)</th>
          <th>petal length (cm)</th>
          <th>petal width (cm)</th>
          <th>fleur</th>
          <th>location</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>5.1</td>
          <td>3.5</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
        <tr>
          <th>1</th>
          <td>4.9</td>
          <td>3.0</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
        <tr>
          <th>2</th>
          <td>4.7</td>
          <td>3.2</td>
          <td>1.3</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
        <tr>
          <th>3</th>
          <td>4.6</td>
          <td>3.1</td>
          <td>1.5</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
        <tr>
          <th>4</th>
          <td>5.0</td>
          <td>3.6</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
        <tr>
          <th>5</th>
          <td>5.4</td>
          <td>3.9</td>
          <td>1.7</td>
          <td>0.4</td>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
        <tr>
          <th>6</th>
          <td>4.6</td>
          <td>3.4</td>
          <td>1.4</td>
          <td>0.3</td>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
        <tr>
          <th>7</th>
          <td>5.0</td>
          <td>3.4</td>
          <td>1.5</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
        <tr>
          <th>8</th>
          <td>4.4</td>
          <td>2.9</td>
          <td>1.4</td>
          <td>0.2</td>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
        <tr>
          <th>9</th>
          <td>4.9</td>
          <td>3.1</td>
          <td>1.5</td>
          <td>0.1</td>
          <td>setosa</td>
          <td>[Maine, Alaska, Quebec]</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    json_text = merged.to_json(orient='records')
    json_text[:200]




.. parsed-literal::
    '[{"sepal length (cm)":5.1,"sepal width (cm)":3.5,"petal length (cm)":1.4,"petal width (cm)":0.2,"fleur":"setosa","location":["Maine","Alaska","Quebec"]},{"sepal length (cm)":4.9,"sepal width (cm)":3.0'



.. code:: ipython3

    df.to_excel("data.xlsx", index=False)

.. code:: ipython3

    dfe = pandas.read_excel("data.xlsx", engine='openpyxl')
    dfe.tail()






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>sepal length (cm)</th>
          <th>sepal width (cm)</th>
          <th>petal length (cm)</th>
          <th>petal width (cm)</th>
          <th>fleur</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>145</th>
          <td>6.7</td>
          <td>3.0</td>
          <td>5.2</td>
          <td>2.3</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>146</th>
          <td>6.3</td>
          <td>2.5</td>
          <td>5.0</td>
          <td>1.9</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>147</th>
          <td>6.5</td>
          <td>3.0</td>
          <td>5.2</td>
          <td>2.0</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>148</th>
          <td>6.2</td>
          <td>3.4</td>
          <td>5.4</td>
          <td>2.3</td>
          <td>virginica</td>
        </tr>
        <tr>
          <th>149</th>
          <td>5.9</td>
          <td>3.0</td>
          <td>5.1</td>
          <td>1.8</td>
          <td>virginica</td>
        </tr>
      </tbody>
    </table>
    </div>



Q6 : le texte, ça prend trop de place, zippons
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: ipython3

    from zipfile import ZipFile
    
    with ZipFile('data.zip', 'w') as myzip:
        myzip.write('data.xlsx')
        myzip.write("2020_json_xml.ipynb")

.. code:: ipython3

    import glob
    glob.glob("*.zip")




.. parsed-literal::
    ['data.zip']



Q7 : que vous inspire `protobuf <https://developers.google.com/protocol-buffers>`__ ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Protobuf est un format de sérialisation au même titre que
`pickle <https://docs.python.org/3/library/pickle.html>`__. La
`sérialisation <https://fr.wikipedia.org/wiki/S%C3%A9rialisation>`__
désigne un procédé qui permet d’enregister tout un tas d’information
sous la forme d’ensemble d’octets contigü. En gros, on enregistre un
ensemble de variables en même quel qu’il soit, dans un unique fichier
contigü.
`json <https://fr.wikipedia.org/wiki/JavaScript_Object_Notation>`__ est
un format utilisé par les outils de sérialisation. Tout est en texte et
la plupart du temps, les nombres réelles prennent moins de place quand
ils sont stockés comme ils sont sockés en mémoire sur 8 octets. C’est le
premier problème résolu par protobuf. Ensuite, le format *json* est très
générique mais il suggère de stocker le nom des colonnes à chaque ligne,
c’est une information qui est dupliqué à chaque ligne… *protobuf*
propose une façon de ne pas les sotcker du tout. Je passe les détails.
Ils reviendront quand le problème de communiquer efficacement des
données se posera.