.. _pigparamsclouderacorrectionrst:

=========================================
PIG et Paramètres (Cloudera) (correction)
=========================================


.. only:: html

    **Links:** :download:`notebook <pig_params_cloudera_correction.ipynb>`, :downloadlink:`html <pig_params_cloudera_correction2html.html>`, :download:`PDF <pig_params_cloudera_correction.pdf>`, :download:`python <pig_params_cloudera_correction.py>`, :downloadlink:`slides <pig_params_cloudera_correction.slides.html>`, :githublink:`GitHub|_doc/notebooks/pig_hive/pig_params_cloudera_correction.ipynb|*`


Correction.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





Connexion au cluster
--------------------

On prend le cluster
`Cloudera <http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx/notebooks/td3a_cenonce_session6.html#p2>`__.
Il faut exécuter ce script pour pouvoir notifier au notebook que la
variable ``params`` existe.

.. code:: ipython3

    import pyensae
    from pyquickhelper.ipythonhelper import open_html_form
    params={"server":"df...fr", "username":"", "password":""}
    open_html_form(params=params,title="server + credentials", key_save="params")






.. raw:: html

    <div style="background-color:gainsboro; padding:2px; border:0px;"><b>server + credentials</b>
    <br />password <input type="password" id="paramspassword" value="" size="80" />
    <br />server <input type="text" id="paramsserver" value="df...fr" size="80" />
    <br />username <input type="text" id="paramsusername" value="" size="80" />
    <br /><button onclick="set_valueparams()">Ok</button></div>
    <script type="text/Javascript">
    function paramscallback(msg) {
       var ret = msg.content.data['text/plain'];
       $('#outparams').text(ret);
    }
    function set_valueparams(){
       command='params = {' ;
       var paramspasswordvar_value = document.getElementById('paramspassword').value;
       command += '"password":"' + paramspasswordvar_value + '",';
       var paramsservervar_value = document.getElementById('paramsserver').value;
       command += '"server":"' + paramsservervar_value + '",';
       var paramsusernamevar_value = document.getElementById('paramsusername').value;
       command += '"username":"' + paramsusernamevar_value + '",';
       command += '}';
       var kernel = IPython.notebook.kernel;
       kernel.execute(command);
    }
    </script>



.. code:: ipython3

    import pyensae
    %load_ext pyensae
    %load_ext pyenbc
    password = params["password"]
    server = params["server"]
    username = params["username"]
    client = %remote_open
    client




.. parsed-literal::
    <pyensae.remote.ssh_remote_connection.ASSHClient at 0x9e20910>



Exercice 1 : min, max
---------------------

On ajoute deux paramètres pour construire l’histogramme entre deux
valeurs ``a``,\ ``b``. Ajouter ces deux paramètres au nom du fichier de
sortie peut paraître raisonnable mais l’interpréteur a du mal à
identifier les paramètres ``Undefined parameter : bins_``. On utilise
des tirets.

.. code:: ipython3

    %%PIG histogramab.pig
    
    values = LOAD 'random/random.sample.txt' USING PigStorage('\t') AS (x:double);
    
    values_f = FILTER values BY x >= $a AND x <= $b ;   -- ligne ajoutée
    
    values_h = FOREACH values_f GENERATE x, ((int)(x / $bins)) * $bins AS h ;
    
    hist_group = GROUP values_h BY h ;
    
    hist = FOREACH hist_group GENERATE group, COUNT(values_h) AS nb ;
    
    STORE hist INTO 'random/histo_$bins-$a-$b.txt' USING PigStorage('\t') ;

.. code:: ipython3

    if client.dfs_exists("random/histo_0.1-0.2-0.8.txt"):
        client.dfs_rm("random/histo_0.1-0.2-0.8.txt", recursive=True)

.. code:: ipython3

    client.pig_submit("histogramab.pig", redirection="redirection", 
                      params =dict(bins="0.1", a="0.2", b="0.8") )




.. parsed-literal::
    ('', '')



.. code:: ipython3

    %remote_cmd tail redirection.err






.. raw:: html

    <pre>
    Spillable Memory Manager spill count : 0
    Total bags proactively spilled: 0
    Total records proactively spilled: 0

    Job DAG:
    job_1414491244634_0189


    2014-12-03 22:55:47,929 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
    2014-12-03 22:55:48,031 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

    </pre>



.. code:: ipython3

    %dfs_ls random






.. raw:: html

    <div style="max-height:1000px;max-width:1500px;overflow:auto;">
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>attributes</th>
          <th>code</th>
          <th>alias</th>
          <th>folder</th>
          <th>size</th>
          <th>date</th>
          <th>time</th>
          <th>name</th>
          <th>isdir</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td> drwxr-xr-x</td>
          <td> -</td>
          <td> xavierdupre</td>
          <td> xavierdupre</td>
          <td>      0</td>
          <td> 2014-12-03</td>
          <td> 22:55</td>
          <td> random/histo_0.1-0.2-0.8.txt</td>
          <td>  True</td>
        </tr>
        <tr>
          <th>1</th>
          <td> drwxr-xr-x</td>
          <td> -</td>
          <td> xavierdupre</td>
          <td> xavierdupre</td>
          <td>      0</td>
          <td> 2014-11-28</td>
          <td> 00:11</td>
          <td>         random/histo_0.1.txt</td>
          <td>  True</td>
        </tr>
        <tr>
          <th>2</th>
          <td> -rw-r--r--</td>
          <td> 3</td>
          <td> xavierdupre</td>
          <td> xavierdupre</td>
          <td> 202586</td>
          <td> 2014-11-27</td>
          <td> 23:38</td>
          <td>     random/random.sample.txt</td>
          <td> False</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    if os.path.exists("histo.txt") : os.remove("histo.txt")
    client.download_cluster("random/histo_0.1-0.2-0.8.txt","histo.txt", merge=True)




.. parsed-literal::
    'random/histo_0.1-0.2-0.8.txt'



.. code:: ipython3

    import matplotlib.pyplot as plt
    plt.style.use('ggplot')
    import pandas
    df = pandas.read_csv("histo.txt", sep="\t",names=["bin","nb"])
    df.plot(x="bin",y="nb",kind="bar")




.. parsed-literal::
    <matplotlib.axes._subplots.AxesSubplot at 0xa0c5c70>




.. image:: pig_params_cloudera_correction_12_1.png