.. _azureblobstoragerst:

=========================================
File Manipulation with Azure Blob Storage
=========================================


.. only:: html

    **Links:** :download:`notebook <azure_blob_storage.ipynb>`, :downloadlink:`html <azure_blob_storage2html.html>`, :download:`PDF <azure_blob_storage.pdf>`, :download:`python <azure_blob_storage.py>`, :downloadlink:`slides <azure_blob_storage.slides.html>`, :githublink:`GitHub|_doc/notebooks/azure_blob_storage.ipynb|*`


We try a few file manipulation between a local computer and a blob
storage on Azure. It requires
`azure-sdk-for-python <https://github.com/Azure/azure-sdk-for-python>`__
and
`pyenbc <http://www.xavierdupre.fr/app/pyenbc/helpsphinx/index.html>`__.
We first create a dummy file.

.. code:: ipython3

    import pandas, random
    mat = [ {"x":random.random(), "y":random.random()} for i in range(0,1000)]
    df = pandas.DataFrame(mat)
    df.to_csv("randomxy.txt", sep="\t", encoding="utf8")

We need credentials and to avoid having them in clear in the notebook,
we use a HTML form:

.. code:: ipython3

    import pyquickhelper.ipythonhelper as ipy
    params={"blob_storage":"hdblobstorage", "password":""}
    ipy.open_html_form(params=params,title="credentials",key_save="blobservice")






.. raw:: html

    <div style="background-color:gainsboro; padding:2px; border:0px;"><b>credentials</b>
    <br />blob_storage <input type="text" id="blobserviceblob_storage" value="hdblobstorage" size="80" />
    <br />password <input type="password" id="blobservicepassword" value="" size="80" />
    <br /><button onclick="set_valueblobservice()">Ok</button></div>
    <script type="text/Javascript">
    function blobservicecallback(msg) {
       var ret = msg.content.data['text/plain'];
       $('#outblobservice').text(ret);
    }
    function set_valueblobservice(){
       command='blobservice = {' ;
       var blobserviceblob_storagevar_value = document.getElementById('blobserviceblob_storage').value;
       command += '"blob_storage":"' + blobserviceblob_storagevar_value + '",';
       var blobservicepasswordvar_value = document.getElementById('blobservicepassword').value;
       command += '"password":"' + blobservicepasswordvar_value + '",';
       command += '}';
       var kernel = IPython.notebook.kernel;
       kernel.execute(command);
    }
    </script>



We stored the values in two variables in the workspace:

.. code:: ipython3

    blobstorage = blobservice["blob_storage"]
    blobpassword = blobservice["password"]

We need pyensae >= 1.2:

.. code:: ipython3

    import pyensae
    import pyenbc
    %load_ext pyensae
    %load_ext pyenbc
    pyensae.__version__, pyenbc.__version__


.. parsed-literal::
    The pyensae extension is already loaded. To reload it, use:
      %reload_ext pyensae
    The pyenbc extension is already loaded. To reload it, use:
      %reload_ext pyenbc




.. parsed-literal::

    '1.2'



.. code:: ipython3

    %blob_open --help


.. parsed-literal::
    usage: blob_open [-h] [-b BLOBSTORAGE] [-p BLOBPASSWORD]
    open a connection to an Azure blob storage, by default, the magic command
    takes blobstorage and blobpassword local variables as default values
    optional arguments:
      -h, --help            show this help message and exit
      -b BLOBSTORAGE, --blobstorage BLOBSTORAGE
                            blob storage name
      -p BLOBPASSWORD, --blobpassword BLOBPASSWORD
                            blob password
    usage: blob_open [-h] [-b BLOBSTORAGE] [-p BLOBPASSWORD]
    


We open a connection to the blob storage:

.. code:: ipython3

    cl, bs = %blob_open
    cl, bs




.. parsed-literal::
    (<pyenbc.remote.azure_connection.AzureClient at 0xa4a2a20>,
     <azure.storage.blob.blobservice.BlobService at 0xa4a2b00>)



We extract the available containers:

.. code:: ipython3

    l = %blob_containers
    l




.. parsed-literal::
    ['clusterensaeazure1',
     'clusterensaeazure2',
     'clusterensaeazure2-1',
     'hdblobstorage',
     'petittest',
     'sparkclus',
     'sparkclus2',
     'testhadoopensae']



We get the content of one container:

.. code:: ipython3

    df = %blob_ls hdblobstorage
    df.tail(n=5)






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>name</th>
          <th>last_modified</th>
          <th>content_type</th>
          <th>content_length</th>
          <th>blob_type</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>4995</th>
          <td>velib_several_days/paris.2014-11-14_15-54-58.6...</td>
          <td>Fri, 28 Nov 2014 10:34:15 GMT</td>
          <td>application/octet-stream</td>
          <td>524941</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>4996</th>
          <td>velib_several_days/paris.2014-11-14_15-55-57.8...</td>
          <td>Fri, 28 Nov 2014 10:34:16 GMT</td>
          <td>application/octet-stream</td>
          <td>524944</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>4997</th>
          <td>velib_several_days/paris.2014-11-14_15-56-58.5...</td>
          <td>Fri, 28 Nov 2014 10:34:17 GMT</td>
          <td>application/octet-stream</td>
          <td>522499</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>4998</th>
          <td>velib_several_days/paris.2014-11-14_15-57-57.8...</td>
          <td>Fri, 28 Nov 2014 10:34:17 GMT</td>
          <td>application/octet-stream</td>
          <td>524958</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>4999</th>
          <td>velib_several_days/paris.2014-11-14_15-58-58.5...</td>
          <td>Fri, 28 Nov 2014 10:34:18 GMT</td>
          <td>application/octet-stream</td>
          <td>523757</td>
          <td>BlockBlob</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    %hd_wasb_prefix




.. parsed-literal::
    'wasb://hdblobstorage@hdblobstorage.blob.core.windows.net/'



.. code:: ipython3

    cl.wasb_to_file("hdblobstorage", "velib_several_days")




.. parsed-literal::
    'wasb://hdblobstorage@hdblobstorage.blob.core.windows.net/velib_several_days'



We upload the file we created in the first cell:

.. code:: ipython3

    %blob_up randomxy.txt clusterensaeazure1/testpyenbc/randomxy.txt




.. parsed-literal::
    'testpyenbc/randomxy.txt'



We check the file is over there:

.. code:: ipython3

    %blob_ls clusterensaeazure1/testpyenbc






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>name</th>
          <th>last_modified</th>
          <th>content_type</th>
          <th>content_length</th>
          <th>blob_type</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>testpyenbc/randomxy.txt</td>
          <td>Sat, 26 Sep 2015 22:05:12 GMT</td>
          <td>application/octet-stream</td>
          <td>43483</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>1</th>
          <td>testpyenbc/randomxy2.txt</td>
          <td>Sat, 26 Sep 2015 21:50:55 GMT</td>
          <td>application/octet-stream</td>
          <td>43456</td>
          <td>BlockBlob</td>
        </tr>
      </tbody>
    </table>
    </div>



We try an extended version:

.. code:: ipython3

    %blob_lsl clusterensaeazure1/testpyenbc






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>blob_type</th>
          <th>content_encoding</th>
          <th>content_language</th>
          <th>content_length</th>
          <th>content_md5</th>
          <th>content_type</th>
          <th>copy_completion_time</th>
          <th>copy_id</th>
          <th>copy_progress</th>
          <th>copy_source</th>
          <th>copy_status</th>
          <th>copy_status_description</th>
          <th>etag</th>
          <th>last_modified</th>
          <th>lease_duration</th>
          <th>lease_state</th>
          <th>lease_status</th>
          <th>name</th>
          <th>url</th>
          <th>xms_blob_sequence_number</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>BlockBlob</td>
          <td></td>
          <td></td>
          <td>43483</td>
          <td></td>
          <td>application/octet-stream</td>
          <td></td>
          <td></td>
          <td></td>
          <td></td>
          <td></td>
          <td></td>
          <td>0x8D2C6BE8D4DEB43</td>
          <td>Sat, 26 Sep 2015 22:05:12 GMT</td>
          <td></td>
          <td>available</td>
          <td>unlocked</td>
          <td>testpyenbc/randomxy.txt</td>
          <td></td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>BlockBlob</td>
          <td></td>
          <td></td>
          <td>43456</td>
          <td></td>
          <td>application/octet-stream</td>
          <td></td>
          <td></td>
          <td></td>
          <td></td>
          <td></td>
          <td></td>
          <td>0x8D2C6BC8E2C38FB</td>
          <td>Sat, 26 Sep 2015 21:50:55 GMT</td>
          <td></td>
          <td>available</td>
          <td>unlocked</td>
          <td>testpyenbc/randomxy2.txt</td>
          <td></td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>



If you need information not accessible through a magic command, you can
use the variable ``bs`` (type
`azure.storage.blobservice.BlobService <http://www.xavierdupre.fr/app/azure-sdk-for-python/helpsphinx/storage/blobservice.html#module-azure.storage.blobservice>`__):

.. code:: ipython3

    l=bs.get_block_list("clusterensaeazure1", "testpyenbc/randomxy.txt")
    for _ in l.committed_blocks:
        print("size=",_.size, "id=",_.id)


.. parsed-literal::
    size= 43483 id= 00000000


We download this again to the local computer:

.. code:: ipython3

    %blob_down clusterensaeazure1/testpyenbc/randomxy.txt randomxx_copy.txt --overwrite




.. parsed-literal::
    'randomxx_copy.txt'



.. code:: ipython3

    %lsr r.*[.]txt






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>directory</th>
          <th>last_modified</th>
          <th>name</th>
          <th>size</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>False</td>
          <td>2015-09-26 23:50:56.776239</td>
          <td>.\randomall.txt</td>
          <td>84.88 Kb</td>
        </tr>
        <tr>
          <th>1</th>
          <td>False</td>
          <td>2015-09-27 00:05:14.546891</td>
          <td>.\randomxx_copy.txt</td>
          <td>42.46 Kb</td>
        </tr>
        <tr>
          <th>2</th>
          <td>False</td>
          <td>2015-09-27 00:04:55.847278</td>
          <td>.\randomxy.txt</td>
          <td>42.46 Kb</td>
        </tr>
      </tbody>
    </table>
    </div>



PIG scripts usually produce more than one output and it is convenient to
merge them while downloading them. To test that, we upload a second time
our file with a different names:

.. code:: ipython3

    %blob_up randomxy.txt clusterensaeazure1/testpyenbc/randomxy2.txt




.. parsed-literal::
    'testpyenbc/randomxy2.txt'



.. code:: ipython3

    %blob_ls clusterensaeazure1/testpyenbc






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>name</th>
          <th>last_modified</th>
          <th>content_type</th>
          <th>content_length</th>
          <th>blob_type</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>testpyenbc/randomxy.txt</td>
          <td>Sat, 26 Sep 2015 22:05:12 GMT</td>
          <td>application/octet-stream</td>
          <td>43483</td>
          <td>BlockBlob</td>
        </tr>
        <tr>
          <th>1</th>
          <td>testpyenbc/randomxy2.txt</td>
          <td>Sat, 26 Sep 2015 22:05:18 GMT</td>
          <td>application/octet-stream</td>
          <td>43483</td>
          <td>BlockBlob</td>
        </tr>
      </tbody>
    </table>
    </div>



And we merge them:

.. code:: ipython3

    %blob_downmerge clusterensaeazure1/testpyenbc randomall.txt --overwrite




.. parsed-literal::
    'randomall.txt'



We check the size of file ``randomall.txt`` is twice bigger:

.. code:: ipython3

    %lsr r.*[.]txt






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>directory</th>
          <th>last_modified</th>
          <th>name</th>
          <th>size</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>False</td>
          <td>2015-09-27 00:05:32.134221</td>
          <td>.\randomall.txt</td>
          <td>84.93 Kb</td>
        </tr>
        <tr>
          <th>1</th>
          <td>False</td>
          <td>2015-09-27 00:05:14.546891</td>
          <td>.\randomxx_copy.txt</td>
          <td>42.46 Kb</td>
        </tr>
        <tr>
          <th>2</th>
          <td>False</td>
          <td>2015-09-27 00:04:55.847278</td>
          <td>.\randomxy.txt</td>
          <td>42.46 Kb</td>
        </tr>
      </tbody>
    </table>
    </div>



We finally remove the files from the blob storage:

.. code:: ipython3

    %blob_delete clusterensaeazure1/testpyenbc/randomxy.txt
    %blob_delete clusterensaeazure1/testpyenbc/randomxy2.txt




.. parsed-literal::
    True



We check it disappeared:

.. code:: ipython3

    %blob_ls clusterensaeazure1/testpyenbc/






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>name</th>
          <th>url</th>
        </tr>
      </thead>
      <tbody>
      </tbody>
    </table>
    </div>



And we close the connection:

.. code:: ipython3

    %blob_close




.. parsed-literal::
    True



**END**