module remote.magic_azure

Inheritance diagram of pyenbc.remote.magic_azure

Short summary

module pyenbc.remote.magic_azure

Magic command to run PIG script with Azure.

source on GitHub

Classes

class truncated documentation
MagicAzure Defines magic commands to access blob storage

Functions

function truncated documentation
register_azure_magics register magics function, can be called from a notebook

Properties

property truncated documentation
Context return the context or None
cross_validation_lock A contextmanager for running a block with our cross validation lock set to True. At the end of the block, …

Static Methods

staticmethod truncated documentation
blob_copy_parser defines the way to parse the magic command %blob_copy
blob_delete_parser defines the way to parse the magic command %blob_delete
blob_down_parser Defines the way to parse the magic command %blob_down.
blob_downmerge_parser defines the way to parse the magic command %blob_downmerge
blob_head_parser defines the way to parse the magic command %blob_head
blob_ls_parser defines the way to parse the magic command %blob_ls
blob_lsl_parser defines the way to parse the magic command %blob_lsl
blob_open_parser defines the way to parse the magic command %blob_open
blob_path_parser defines the magic command %blob_path, checks the path used in commands blob_down, blob_up
blob_rmr_parser defines the way to parse the magic command %blob_rmr
blob_up_parser Defines the way to parse the magic command %blob_up.
hd_job_kill_parser defines the way to parse the magic command %hd_job_kill
hd_job_status_parser defines the way to parse the magic command %hd_job_status
hd_open_parser defines the way to parse the magic command %hd_open
hd_pig_submit_parser Defines the way to parse the magic command %hd_pig_submit.
hd_queue_parser defines the way to parse the magic command %hd_queue
hd_tail_stderr_parser defines the way to parse the magic command %hd_tail_stderr
HIVE_azure_parser defines the way to parse the magic command %HIVE_azure
HIVE_azure_submit_parser Defines the way to parse the magic command %HIVE_azure_submit.
jython_parser defines the way to parse the magic command %%jython
PIG_azure_parser defines the way to parse the magic command %%PIG_azure
runjython_parser defines the way to parse the magic command %%runjython

Methods

method truncated documentation
_interpret_path interpret a path
_replace_params replaces parameter such __PASSWORD__ by variable in the notebook environment
_run_jython run a jython script
azureclient returns the AzureClient object
blob_close close the connection and remove the connection from the notebook workspace
blob_containers returns the list of containers
blob_copy copy a blob storage, see l-magic-path-container
blob_delete deletes a blob, see l-magic-path-container
blob_down download a file from the blob storage, see l-magic-path-container Example
blob_downmerge download all files from a folder, see l-magic-path-container Example
blob_head download a file from the blob storage and display its head, see l-magic-path-container Example
blob_ls defines command %blob_ls, see l-magic-path-container
blob_lsl defines command %blob_lsl (extended version of blob_lsl), see l-magic-path-container
blob_open  
blob_path checks the path used in commands blob_down, blob_up, see _interpret_path(), l-magic-path-container
blob_rm calls blob_delete()
blob_rmr deletes a folder, see l-magic-path-container
blob_up upload a file to the blob storage, we assume the container is the first element of the path, see l-magic-path-container
blobcontainer returns the Blob Storage container
blobservice returns the BlobService object
create_client Create a AzureClient and stores in the workspace.
get_blob_connection returns the connection stored in the workspace
hd_job_kill defines %hd_job_kill
hd_job_status defines %hd_job_status
hd_open Opens a connection to blob service.
hd_pig_submit Defines command %hd_pig_submit.
hd_queue defines %hd_queue
hd_tail_stderr defines %hd_tail_stderr
hd_wasb_prefix defines %hd_wasb_prefix, returns the prefix used to connect to the blob storage, it includes the container
HIVE_azure defines command %%HIVE_azure
HIVE_azure_submit Defines command %HIVE_azure_submit.
jython Defines command %%runjython. run a jython script used for streaming in HDInsight, the function appends …
PIG_azure defines command %%PIG_azure
runjpython Defines command %%runjython.

Documentation

Magic command to run PIG script with Azure.

source on GitHub

class pyenbc.remote.magic_azure.MagicAzure(shell=None, **kwargs)[source]

Bases: pyquickhelper.ipythonhelper.magic_class.MagicClassWithHelpers

Defines magic commands to access blob storage and HDInsight.

When the container is not specified, it will take the default one.

Magic command %blob_open does not work

Try this:

%load_ext pyenbc

The exception tells more about what goes wrong. Usually a module is missing.

Incorrect padding

The following crypted message happens sometimes:

Error: Incorrect padding

It is usually due to an incorrect password. Some notebooks uses:

import pyquickhelper.ipythonhelper as ipy
params={"blob_storage":"hdblobstorage", "password":""}
ipy.open_html_form(params=params,title="credentials",key_save="blobservice")

blobstorage = blobservice["blob_storage"]
blobpassword = blobservice["password"]

%load_ext pyenbc
%blob_open

This code avoids the author letting password in a notebook but you can just replace everything by:

blobstorage = "<username>"
blobpassword = "****long*key*******=="

%load_ext pyenbc
%blob_open

source on GitHub

HIVE_azure(line, cell=None)[source]

defines command %%HIVE_azure

HIVE_azure

The code for magic command %HIVE_azure is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

source on GitHub

static HIVE_azure_parser()[source]

defines the way to parse the magic command %HIVE_azure

source on GitHub

HIVE_azure_submit(line)[source]

Defines command %HIVE_azure_submit.

HIVE_azure_submit

The code for magic command %HIVE_azure_submit is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.hive_submit(bs, cl.account_name, hive_file_name, dependencies, **options)

source on GitHub

static HIVE_azure_submit_parser()[source]

Defines the way to parse the magic command %HIVE_azure_submit.

source on GitHub

PIG_azure(line, cell=None)[source]

defines command %%PIG_azure

PIG_azure

The code for magic command %PIG_azure is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

source on GitHub

static PIG_azure_parser()[source]

defines the way to parse the magic command %%PIG_azure

source on GitHub

_interpret_path(line, cl, bs, empty_is_value=False)[source]

interpret a path

Parameters:
  • line – line (see l-magic-path-container)
  • clAzureClient
  • bs – blob service
  • empty_is_value – if True, do not raise an exception
Returns:

container, remotepath

source on GitHub

_replace_params(cell)[source]

replaces parameter such __PASSWORD__ by variable in the notebook environment

Parameters:cell – string
Returns:modified string

source on GitHub

_run_jython(cell, filename, func_name, args, true_jython=None)[source]

run a jython script

Parameters:
  • cell – content of the cell
  • filename – filename used to store the content of the cell
  • func_name – function name
  • args – list of arguments to run
  • true_jython – jython (True) or this Python (False)
Returns:

out, err

source on GitHub

_trait_default_generators = {}
azureclient(line)[source]

returns the AzureClient object

source on GitHub

blob_close(line)[source]

close the connection and remove the connection from the notebook workspace

blob_close

Does nothing.

source on GitHub

blob_containers(line)[source]

returns the list of containers

source on GitHub

blob_copy(line)[source]

copy a blob storage, see l-magic-path-container

blob_copy

The code for magic command %blob_copy is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.copy_blob(bs, container, dest, src)

source on GitHub

static blob_copy_parser()[source]

defines the way to parse the magic command %blob_copy

source on GitHub

blob_delete(line)[source]

deletes a blob, see l-magic-path-container

blob_delete

The code for magic command %blob_delete is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.delete_blob(bs, container, remotepath)

source on GitHub

static blob_delete_parser()[source]

defines the way to parse the magic command %blob_delete

source on GitHub

blob_down(line)[source]

download a file from the blob storage, see l-magic-path-container

Example:

%blob_down remotepath localfile

the command does not allow spaces in file names

blob_down

The code for magic command %blob_down is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.download(bs, container, remotepath, localfile)

source on GitHub

static blob_down_parser()[source]

Defines the way to parse the magic command %blob_down.

source on GitHub

blob_downmerge(line)[source]

download all files from a folder, see l-magic-path-container

Example:

%blob_downmerge remotepath localfile

the command does not allow spaces in file names

blob_downmerge

The code for magic command %blob_downmerge is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.download_merge(bs, container, remotepath, localfile)

New in version 1.1.

source on GitHub

static blob_downmerge_parser()[source]

defines the way to parse the magic command %blob_downmerge

source on GitHub

blob_head(line)[source]

download a file from the blob storage and display its head, see l-magic-path-container

Example:

%blob_head remotepath

the command does not allow spaces in file names

blob_head

The code for magic command %blob_head is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.df_head(bs, container, remotepath, localfile)

source on GitHub

static blob_head_parser()[source]

defines the way to parse the magic command %blob_head

source on GitHub

blob_ls(line)[source]

defines command %blob_ls, see l-magic-path-container

blob_ls

The code for magic command %blob_ls is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.ls(bs, container, remotepath)

source on GitHub

static blob_ls_parser()[source]

defines the way to parse the magic command %blob_ls

source on GitHub

blob_lsl(line)[source]

defines command %blob_lsl (extended version of blob_lsl), see l-magic-path-container

blob_lsl

The code for magic command %blob_lsl is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.ls(bs, container, remotepath, add_metadata=True)

source on GitHub

static blob_lsl_parser()[source]

defines the way to parse the magic command %blob_lsl

source on GitHub

blob_open(line)[source]

blob_open

Opens a connection to blob service. It returns objects AzureClient and BlobService.

The code for magic command %blob_open is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()

Changed in version 1.1.

source on GitHub

static blob_open_parser()[source]

defines the way to parse the magic command %blob_open

source on GitHub

blob_path(line)[source]

checks the path used in commands blob_down, blob_up, see _interpret_path(), l-magic-path-container

blob_path

The code for magic command %blob_path is equivalent to:

if line.startswith("/"):
    container = account_name
    remotepath = remotepath.lstrip("/")
else:
    spl = line.split("/")
    container = spl[0]
    remotepath = None if len(spl) == 1 else "/".join(spl[1:])

result = container, remotepath

source on GitHub

static blob_path_parser()[source]

defines the magic command %blob_path, checks the path used in commands blob_down, blob_up

source on GitHub

blob_rm(line)[source]

calls blob_delete

New in version 1.1.

source on GitHub

blob_rmr(line)[source]

deletes a folder, see l-magic-path-container

blob_rmr

The code for magic command %blob_rmr is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.delete_folder(bs, container, remotepath)

source on GitHub

static blob_rmr_parser()[source]

defines the way to parse the magic command %blob_rmr

source on GitHub

blob_up(line)[source]

upload a file to the blob storage, we assume the container is the first element of the path, see l-magic-path-container

Example:

%blob_up localfile remotepath

the command does not allow spaces in files

blob_up

The code for magic command %blob_up is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.upload(bs, container, remotepath, localfile)

source on GitHub

static blob_up_parser()[source]

Defines the way to parse the magic command %blob_up.

source on GitHub

blobcontainer(line)[source]

returns the Blob Storage container

source on GitHub

blobservice(line)[source]

returns the BlobService object

source on GitHub

create_client(account_name, account_key, hadoop_server=None, hadoop_password=None, username=None)[source]

Create a AzureClient and stores in the workspace.

Parameters:
  • account_name – login
  • account_key – password
  • hadoop_server – hadoop server
  • hadoop_password – hadoop password
  • username – username
Returns:

instance of AzureClient

source on GitHub

get_blob_connection()[source]

returns the connection stored in the workspace

source on GitHub

hd_job_kill(line)[source]

defines %hd_job_kill

hd_job_kill

The code for magic command %hd_job_kill is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_kill(jobid)

source on GitHub

static hd_job_kill_parser()[source]

defines the way to parse the magic command %hd_job_kill

source on GitHub

hd_job_status(line)[source]

defines %hd_job_status

hd_job_status

The code for magic command %hd_job_status is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_status(jobid)

source on GitHub

static hd_job_status_parser()[source]

defines the way to parse the magic command %hd_job_status

source on GitHub

hd_open(line)[source]

Opens a connection to blob service.

hd_open

Opens a connection to blob service. It returns objects AzureClient and BlobService.

The code for magic command %hd_open is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()

source on GitHub

static hd_open_parser()[source]

defines the way to parse the magic command %hd_open

source on GitHub

hd_pig_submit(line)[source]

Defines command %hd_pig_submit.

hd_pig_submit

The code for magic command %hd_pig_submit is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.pig_submit(bs, cl.account_name, pig_file_name, dependencies, **options)

source on GitHub

static hd_pig_submit_parser()[source]

Defines the way to parse the magic command %hd_pig_submit.

source on GitHub

hd_queue(line)[source]

defines %hd_queue

hd_queue

The code for magic command %hd_queue is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_queue(showall=showall)

source on GitHub

static hd_queue_parser()[source]

defines the way to parse the magic command %hd_queue

source on GitHub

hd_tail_stderr(line)[source]

defines %hd_tail_stderr

Warning

This function gets the status of the job to get the script name. But the rediction uses the script name and not the job id. As a consequence, if the same script name is run multiple times, the redirection will contain the output of multiples jobs.

hd_tail_stderr

The code for magic command %hd_tail_stderr is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.standard_outputs(job_id, bs, cl.account_name, ".")

source on GitHub

static hd_tail_stderr_parser()[source]

defines the way to parse the magic command %hd_tail_stderr

source on GitHub

hd_wasb_prefix(line)[source]

defines %hd_wasb_prefix, returns the prefix used to connect to the blob storage, it includes the container name

source on GitHub

jython(line, cell=None)[source]

Defines command %%runjython.

run a jython script used for streaming in HDInsight, the function appends fake decorator a timeout is set up at 10s

The magic function create another file included the decoration. It runs the script with Jython (see the default version)

See In a python script how can I ignore Apache Pig’s Python Decorators for standalone unit testing.

New in version 1.1.

source on GitHub

static jython_parser()[source]

defines the way to parse the magic command %%jython

source on GitHub

runjpython(line, cell=None)[source]

Defines command %%runjython.

runjpython

Run a jython script used for streaming in HDInsight, the function appends fake decorator a timeout is set up at 10s

The magic function create another file included the decoration. It runs the script with this version of Python.

See In a python script how can I ignore Apache Pig’s Python Decorators for standalone unit testing

See _run_jython to see the code.

New in version 1.1.

source on GitHub

static runjython_parser()[source]

defines the way to parse the magic command %%runjython

source on GitHub

pyenbc.remote.magic_azure.register_azure_magics(ip=None)[source]

register magics function, can be called from a notebook

Parameters:ip – from get_ipython()

source on GitHub