.. _pigazurecorrectionrst:

==========================================
Map/Reduce avec PIG sur Azure - correction
==========================================


.. only:: html

    **Links:** :download:`notebook <pig_azure_correction.ipynb>`, :downloadlink:`html <pig_azure_correction2html.html>`, :download:`PDF <pig_azure_correction.pdf>`, :download:`python <pig_azure_correction.py>`, :downloadlink:`slides <pig_azure_correction.slides.html>`, :githublink:`GitHub|_doc/notebooks/pig_hive/pig_azure_correction.ipynb|*`


.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





Données
-------

On considère le jeu de données suivant : `Localization Data for Person
Activity Data
Set <https://archive.ics.uci.edu/ml/datasets/Localization+Data+for+Person+Activity>`__
qu’on récupère comme indiqué dans le notebook de l’énoncé.

.. code:: ipython3

    from pyquickhelper.ipythonhelper import open_html_form
    params={"blob_storage":"", "password1":"", "hadoop_server":"", "password2":"", "username":"xavierdupre"}
    open_html_form(params=params,title="server + hadoop + credentials", key_save="blobhp")






.. raw:: html

    <div style="background-color:gainsboro; padding:2px; border:0px;"><b>server + hadoop + credentials</b>
    <br />blob_storage <input type="text" id="blobhpblob_storage" value="" size="80" />
    <br />hadoop_server <input type="text" id="blobhphadoop_server" value="" size="80" />
    <br />password1 <input type="password" id="blobhppassword1" value="" size="80" />
    <br />password2 <input type="password" id="blobhppassword2" value="" size="80" />
    <br />username <input type="text" id="blobhpusername" value="xavierdupre" size="80" />
    <br /><button onclick="set_valueblobhp()">Ok</button></div>
    <script type="text/Javascript">
    function blobhpcallback(msg) {
       var ret = msg.content.data['text/plain'];
       $('#outblobhp').text(ret);
    }
    function set_valueblobhp(){
       command='blobhp = {' ;
       var blobhpblob_storagevar_value = document.getElementById('blobhpblob_storage').value;
       command += '"blob_storage":"' + blobhpblob_storagevar_value + '",';
       var blobhphadoop_servervar_value = document.getElementById('blobhphadoop_server').value;
       command += '"hadoop_server":"' + blobhphadoop_servervar_value + '",';
       var blobhppassword1var_value = document.getElementById('blobhppassword1').value;
       command += '"password1":"' + blobhppassword1var_value + '",';
       var blobhppassword2var_value = document.getElementById('blobhppassword2').value;
       command += '"password2":"' + blobhppassword2var_value + '",';
       var blobhpusernamevar_value = document.getElementById('blobhpusername').value;
       command += '"username":"' + blobhpusernamevar_value + '",';
       command += '}';
       var kernel = IPython.notebook.kernel;
       kernel.execute(command);
    }
    </script>



.. code:: ipython3

    blobstorage = blobhp["blob_storage"]
    blobpassword = blobhp["password1"]
    hadoop_server = blobhp["hadoop_server"]
    hadoop_password = blobhp["password2"]
    username = blobhp["username"]

.. code:: ipython3

    import pyensae
    %load_ext pyensae
    %load_ext pyenbc
    %hd_open




.. parsed-literal::
    (<pyensae.remote.azure_connection.AzureClient at 0xafe7e10>,
     <azure.storage.blob.blobservice.BlobService at 0xafe7e48>)



Exercice 1 : GROUP BY
---------------------

.. code:: ipython3

    import pandas, sqlite3
    con = sqlite3.connect("ConfLongDemo_JSI.db3")
    df = pandas.read_sql("""SELECT activity, count(*) as nb FROM person GROUP BY activity""", con)
    con.close()
    df.head()






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>activity</th>
          <th>nb</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>falling</td>
          <td>2973</td>
        </tr>
        <tr>
          <th>1</th>
          <td>lying</td>
          <td>54480</td>
        </tr>
        <tr>
          <th>2</th>
          <td>lying down</td>
          <td>6168</td>
        </tr>
        <tr>
          <th>3</th>
          <td>on all fours</td>
          <td>5210</td>
        </tr>
        <tr>
          <th>4</th>
          <td>sitting</td>
          <td>27244</td>
        </tr>
      </tbody>
    </table>
    </div>



On vérifie que le fichier qu’on veut traiter est bien là :

.. code:: ipython3

    %blob_ls /testensae/ConfLongDemo_JSI.small.txt






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>name</th>
          <th>last_modified</th>
          <th>content_type</th>
          <th>content_length</th>
          <th>blob_type</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>testensae/ConfLongDemo_JSI.small.txt</td>
          <td>Thu, 29 Oct 2015 00:23:00 GMT</td>
          <td>application/octet-stream</td>
          <td>132727</td>
          <td>BlockBlob</td>
        </tr>
      </tbody>
    </table>
    </div>



Il faut maintenant le faire avec PIG.

.. code:: ipython3

    %%PIG_azure solution_groupby.pig
    
    myinput = LOAD '$CONTAINER/testensae/ConfLongDemo_JSI.small.txt' 
              using PigStorage(',') 
              AS (index:long, sequence, tag, timestamp:long, dateformat, x:double,y:double, z:double, activity) ;
    
    gr = GROUP myinput BY activity ;
    avgact = FOREACH gr GENERATE group, COUNT(myinput) ; 
    
    STORE avgact INTO '$CONTAINER/$PSEUDO/testensae/ConfLongDemo_JSI.small.group.2015.txt' USING PigStorage() ;

On soumet le job :

.. code:: ipython3

    jid = %hd_pig_submit solution_groupby.pig
    jid




.. parsed-literal::
    {'id': 'job_1445989166328_0009'}



On vérifie le status du job :

.. code:: ipython3

    st = %hd_job_status jid["id"]
    st["id"],st["percentComplete"],st["completed"],st["status"]["jobComplete"],st["status"]["state"]




.. parsed-literal::
    ('job_1445989166328_0009', '100% complete', None, False, 'RUNNING')



On regarde si la compilation s’est bien passée :

.. code:: ipython3

    %hd_tail_stderr jid["id"]






.. raw:: html

    <pre>

    Job DAG:
    job_1445989166328_0010


    2015-10-29 00:55:14,395 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://headnodehost:8188/ws/v1/timeline/
    2015-10-29 00:55:14,395 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at headnodehost/100.89.100.164:9010
    2015-10-29 00:55:14,395 [main] INFO  org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at headnodehost/100.89.100.164:10200
    2015-10-29 00:55:14,473 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
    2015-10-29 00:55:14,676 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://headnodehost:8188/ws/v1/timeline/
    2015-10-29 00:55:14,676 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at headnodehost/100.89.100.164:9010
    2015-10-29 00:55:14,676 [main] INFO  org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at headnodehost/100.89.100.164:10200
    2015-10-29 00:55:14,754 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
    2015-10-29 00:55:14,957 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://headnodehost:8188/ws/v1/timeline/
    2015-10-29 00:55:14,957 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at headnodehost/100.89.100.164:9010
    2015-10-29 00:55:14,957 [main] INFO  org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at headnodehost/100.89.100.164:10200
    2015-10-29 00:55:15,020 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
    2015-10-29 00:55:15,082 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
    2015-10-29 00:55:15,113 [main] INFO  org.apache.pig.Main - Pig script completed in 49 seconds and 706 milliseconds (49706 ms)

    </pre><br />



On regarde le contenu du répertoire sur le blob storage :

.. code:: ipython3

    df=%blob_ls /$PSEUDO/testensae
    list(df["name"])




.. parsed-literal::
    ['xavierdupre/testensae',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.2015.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.2015.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.2015.txt/part-r-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.join.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.join.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.join.txt/part-r-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.txt/part-r-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.keep_walking.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.keep_walking.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.keep_walking.txt/part-m-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking2015.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking2015.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking2015.txt/part-m-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking_2015.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking_2015.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking_2015.txt/part-m-00000']



.. code:: ipython3

    import os
    if os.path.exists("results.group.2015.xt") : os.remove("results.group.2015.txt")
    %blob_downmerge /$PSEUDO/testensae/ConfLongDemo_JSI.small.group.2015.txt results.group.2015.txt




.. parsed-literal::
    'results.group.2015.txt'



.. code:: ipython3

    %lsr res.*[.]txt






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>directory</th>
          <th>last_modified</th>
          <th>name</th>
          <th>size</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>False</td>
          <td>2015-10-29 01:56:11.025867</td>
          <td>.\results.group.2015.txt</td>
          <td>89</td>
        </tr>
        <tr>
          <th>1</th>
          <td>False</td>
          <td>2015-10-29 01:46:45.425028</td>
          <td>.\results.txt</td>
          <td>21.65 Kb</td>
        </tr>
        <tr>
          <th>2</th>
          <td>False</td>
          <td>2015-10-29 01:46:46.705466</td>
          <td>.\results_allfiles.txt</td>
          <td>21.65 Kb</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    %head results.group.2015.txt






.. raw:: html

    <pre>
    lying	267
    falling	30
    sitting	435
    walking	170
    sitting down	56
    standing up from sitting	42

    </pre>



Exercice 2 : JOIN
-----------------

.. code:: ipython3

    con = sqlite3.connect("ConfLongDemo_JSI.db3")
    df = pandas.read_sql("""SELECT person.*, A.nb FROM person INNER JOIN (
                                SELECT activity, count(*) as nb FROM person GROUP BY activity) AS A
                                ON person.activity == A.activity""", con)
    con.close()
    df.head()






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>index</th>
          <th>sequence</th>
          <th>tag</th>
          <th>timestamp</th>
          <th>dateformat</th>
          <th>x</th>
          <th>y</th>
          <th>z</th>
          <th>activity</th>
          <th>nb</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0</td>
          <td>A01</td>
          <td>010-000-024-033</td>
          <td>633790226051280329</td>
          <td>27.05.2009 14:03:25:127</td>
          <td>4.062931</td>
          <td>1.892434</td>
          <td>0.507425</td>
          <td>walking</td>
          <td>32710</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1</td>
          <td>A01</td>
          <td>020-000-033-111</td>
          <td>633790226051820913</td>
          <td>27.05.2009 14:03:25:183</td>
          <td>4.291954</td>
          <td>1.781140</td>
          <td>1.344495</td>
          <td>walking</td>
          <td>32710</td>
        </tr>
        <tr>
          <th>2</th>
          <td>2</td>
          <td>A01</td>
          <td>020-000-032-221</td>
          <td>633790226052091205</td>
          <td>27.05.2009 14:03:25:210</td>
          <td>4.359101</td>
          <td>1.826456</td>
          <td>0.968821</td>
          <td>walking</td>
          <td>32710</td>
        </tr>
        <tr>
          <th>3</th>
          <td>3</td>
          <td>A01</td>
          <td>010-000-024-033</td>
          <td>633790226052361498</td>
          <td>27.05.2009 14:03:25:237</td>
          <td>4.087835</td>
          <td>1.879999</td>
          <td>0.466983</td>
          <td>walking</td>
          <td>32710</td>
        </tr>
        <tr>
          <th>4</th>
          <td>4</td>
          <td>A01</td>
          <td>010-000-030-096</td>
          <td>633790226052631792</td>
          <td>27.05.2009 14:03:25:263</td>
          <td>4.324462</td>
          <td>2.072460</td>
          <td>0.488065</td>
          <td>walking</td>
          <td>32710</td>
        </tr>
      </tbody>
    </table>
    </div>



Idem, maintenant il faut le faire avec PIG.

.. code:: ipython3

    %%PIG_azure solution_groupby_join.pig
    
    myinput = LOAD '$CONTAINER/testensae/ConfLongDemo_JSI.small.txt' 
              using PigStorage(',') 
              AS (index:long, sequence, tag, timestamp:long, dateformat, x:double,y:double, z:double, activity) ;
    
    gr = GROUP myinput BY activity ;
    avgact = FOREACH gr GENERATE group, COUNT(myinput) ; 
    
    joined = JOIN myinput BY activity, avgact BY group ;
    
    STORE joined INTO '$CONTAINER/$PSEUDO/testensae/ConfLongDemo_JSI.small.group.join.2015.txt' USING PigStorage() ;

.. code:: ipython3

    jid = %hd_pig_submit solution_groupby_join.pig
    jid




.. parsed-literal::
    {'id': 'job_1445989166328_0011'}



.. code:: ipython3

    st = %hd_job_status jid["id"]
    st["id"],st["percentComplete"],st["completed"],st["status"]["jobComplete"],st["status"]["state"], st["userargs"]["file"]




.. parsed-literal::
    ('job_1445989166328_0011',
     '100% complete',
     'done',
     True,
     'SUCCEEDED',
     'wasb://hdblobstorage@hdblobstorage.blob.core.windows.net/xavierdupre/scripts/pig/solution_groupby_join.pig')



.. code:: ipython3

    df=%blob_ls /$PSEUDO/testensae
    df






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>name</th>
          <th>last_modified</th>
          <th>content_type</th>
          <th>content_length</th>
          <th>blob_type</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>xavierdupre/testensae</td>
          <td>Tue, 25 Nov 2014 00:50:34 GMT</td>
          <td>application/octet-stream</td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Thu, 29 Oct 2015 00:55:09 GMT</td>
          <td></td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>2</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Thu, 29 Oct 2015 00:55:09 GMT</td>
          <td>application/octet-stream</td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>3</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Thu, 29 Oct 2015 00:55:08 GMT</td>
          <td>application/octet-stream</td>
          <td>89</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>4</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Thu, 29 Oct 2015 00:58:43 GMT</td>
          <td></td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>5</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Thu, 29 Oct 2015 00:58:43 GMT</td>
          <td>application/octet-stream</td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>6</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Thu, 29 Oct 2015 00:58:42 GMT</td>
          <td>application/octet-stream</td>
          <td>144059</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>7</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Tue, 25 Nov 2014 01:16:11 GMT</td>
          <td></td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>8</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Tue, 25 Nov 2014 01:16:11 GMT</td>
          <td>application/octet-stream</td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>9</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Tue, 25 Nov 2014 01:16:10 GMT</td>
          <td>application/octet-stream</td>
          <td>144059</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>10</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Tue, 25 Nov 2014 01:12:49 GMT</td>
          <td></td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>11</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Tue, 25 Nov 2014 01:12:49 GMT</td>
          <td>application/octet-stream</td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>12</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.g...</td>
          <td>Tue, 25 Nov 2014 01:12:49 GMT</td>
          <td>application/octet-stream</td>
          <td>89</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>13</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.k...</td>
          <td>Tue, 25 Nov 2014 00:50:45 GMT</td>
          <td></td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>14</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.k...</td>
          <td>Tue, 25 Nov 2014 00:50:46 GMT</td>
          <td>application/octet-stream</td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>15</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.k...</td>
          <td>Tue, 25 Nov 2014 00:50:45 GMT</td>
          <td>application/octet-stream</td>
          <td>22166</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>16</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.w...</td>
          <td>Thu, 29 Oct 2015 00:28:30 GMT</td>
          <td></td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>17</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.w...</td>
          <td>Thu, 29 Oct 2015 00:28:30 GMT</td>
          <td>application/octet-stream</td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>18</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.w...</td>
          <td>Thu, 29 Oct 2015 00:28:30 GMT</td>
          <td>application/octet-stream</td>
          <td>22166</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>19</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.w...</td>
          <td>Thu, 29 Oct 2015 00:46:05 GMT</td>
          <td></td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>20</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.w...</td>
          <td>Thu, 29 Oct 2015 00:46:05 GMT</td>
          <td>application/octet-stream</td>
          <td>0</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>21</th>
          <td>xavierdupre/testensae/ConfLongDemo_JSI.small.w...</td>
          <td>Thu, 29 Oct 2015 00:46:04 GMT</td>
          <td>application/octet-stream</td>
          <td>22166</td>
          <td>BlockBlob</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    set(df.name)




.. parsed-literal::
    {'xavierdupre/testensae',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.2015.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.2015.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.2015.txt/part-r-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.join.2015.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.join.2015.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.join.2015.txt/part-r-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.join.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.join.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.join.txt/part-r-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.group.txt/part-r-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.keep_walking.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.keep_walking.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.keep_walking.txt/part-m-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking2015.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking2015.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking2015.txt/part-m-00000',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking_2015.txt',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking_2015.txt/_SUCCESS',
     'xavierdupre/testensae/ConfLongDemo_JSI.small.walking_2015.txt/part-m-00000'}



.. code:: ipython3

    if os.path.exists("results.join.2015.txt") : os.remove("results.join.2015.txt")
    %blob_downmerge /$PSEUDO/testensae/ConfLongDemo_JSI.small.group.join.2015.txt results.join.2015.txt




.. parsed-literal::
    'results.join.2015.txt'



.. code:: ipython3

    %head results.join.2015.txt






.. raw:: html

    <pre>
    999	A01	010-000-024-033	633790226379871138	27.05.2009 14:03:57:987	3.198556661605835	1.1257659196853638	0.3567752242088318	lying	lying	267
    998	A01	020-000-032-221	633790226379600847	27.05.2009 14:03:57:960	4.3730292320251465	1.3821170330047607	0.38861045241355896	lying	lying	267
    997	A01	020-000-033-111	633790226379330550	27.05.2009 14:03:57:933	4.7574005126953125	1.285519003868103	-0.08946932852268219	lying	lying	267
    996	A01	010-000-030-096	633790226379060251	27.05.2009 14:03:57:907	3.182415008544922	1.1020996570587158	0.29104289412498474	lying	lying	267
    995	A01	010-000-024-033	633790226378789954	27.05.2009 14:03:57:880	3.0784008502960205	1.0197675228118896	0.6061218976974487	lying	lying	267
    994	A01	020-000-032-221	633790226378519655	27.05.2009 14:03:57:853	4.36382532119751	1.4307395219802856	0.3206148743629456	lying	lying	267
    993	A01	010-000-024-033	633790226377708776	27.05.2009 14:03:57:770	3.0621800422668457	1.0790562629699707	0.6795752048492432	lying	lying	267
    992	A01	020-000-032-221	633790226377438480	27.05.2009 14:03:57:743	4.371500492095946	1.4781558513641355	0.5384233593940735	lying	lying	267
    991	A01	020-000-033-111	633790226377168187	27.05.2009 14:03:57:717	4.918898105621338	1.1530661582946775	0.19635945558547974	lying	lying	267
    990	A01	010-000-030-096	633790226376897895	27.05.2009 14:03:57:690	3.208510637283325	1.1156394481658936	0.3381773829460144	lying	lying	267

    </pre>



.. raw:: html

   <h3 id="prol">

Prolongements

.. raw:: html

   </h3>

`PIG <http://pig.apache.org/>`__ n’est pas la seule façon d’exécuter des
jobs Map/Reduce. `Hive <https://hive.apache.org/>`__ est un langage dont
la syntaxe est très proche de celle du SQL. L’article `Comparing Pig
Latin and SQL for Constructing Data Processing
Pipelines <https://developer.yahoo.com/blogs/hadoop/comparing-pig-latin-sql-constructing-data-processing-pipelines-444.html>`__
explicite les différences des deux approches.

**langage haut niveau**

Ce qu’il faut retenir est que le langage PIG est un langage haut niveau.
Le programme est compilé en une séquence d’opérations Map/Reduce
transparente pour l’utilisateur. Le temps de développement est très
réduit lorsqu’on le compare au même programme écrit en Java. Le
compilateur construit un plan d’exécution (`quelques exemples
ici <http://chimera.labs.oreilly.com/books/1234000001811/ch07.html#explain>`__)
et infère le nombre de machines requises pour distribuer le job. Cela
suffit pour la plupart des besoins, cela nécessite.

**petits jeux**

Certains jobs peuvent durer des heures, il est conseillée de les essayer
sur des petits jeux de données avant de les faire tourner sur les vrais
données. Il est toujours frustrant de s’apercevoir qu’un job a planté au
bout de deux heures car une chaîne de caractères est vide et que ce cas
n’a pas été prévu.

Avec ces petits jeux, il est possible de faire tourner et conseillé de
tester le job d’abord sur la passerelle (`exécution
local <http://archive.cloudera.com/cdh/3/pig/tutorial.html#Running+the+Pig+Scripts+in+Local+Mode>`__)
avant de le lancer sur le cluster. Avec pyensae, il faut ajouter
l’option ``-local`` à la commande
`hd_pig_submit <http://www.xavierdupre.fr/app/pyensae/helpsphinx/pyensae/remote/magic_azure.html?highlight=hd_pig_submit#pyensae.remote.magic_azure.MagicAzure.hd_pig_submit>`__.

**concaténer les fichiers divisés**

Un programme PIG ne produit pas un fichier mais plusieurs fichiers dans
un répertoire. La commande
`getmerge <http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/FileSystemShell.html>`__
télécharge ces fichiers sur la passerelle et les fusionne en un seul.

**ordre des lignes**

Les jobs sont distribués, même en faisant rien (LOAD + STORE), il n’est
pas garanti que l’ordre des lignes soit préservé. La probabilié que ce
soit le cas est quasi nulle.