module remote.magic_azure

Inheritance diagram of pyenbc.remote.magic_azure

Short summary

module pyenbc.remote.magic_azure

Magic command to run PIG script with Azure.

source on GitHub

Classes

class

truncated documentation

MagicAzure

Defines magic commands to access blob storage

Functions

function

truncated documentation

register_azure_magics

register magics function, can be called from a notebook

Properties

property

truncated documentation

Context

return the context or None

cross_validation_lock

A contextmanager for running a block with our cross validation lock set to True. At the end of the block, …

Static Methods

staticmethod

truncated documentation

blob_copy_parser

defines the way to parse the magic command %blob_copy

blob_delete_parser

defines the way to parse the magic command %blob_delete

blob_down_parser

Defines the way to parse the magic command %blob_down.

blob_downmerge_parser

defines the way to parse the magic command %blob_downmerge

blob_head_parser

defines the way to parse the magic command %blob_head

blob_ls_parser

defines the way to parse the magic command %blob_ls

blob_lsl_parser

defines the way to parse the magic command %blob_lsl

blob_open_parser

defines the way to parse the magic command %blob_open

blob_path_parser

defines the magic command %blob_path, checks the path used in commands blob_down, blob_up

blob_rmr_parser

defines the way to parse the magic command %blob_rmr

blob_up_parser

Defines the way to parse the magic command %blob_up.

hd_job_kill_parser

defines the way to parse the magic command %hd_job_kill

hd_job_status_parser

defines the way to parse the magic command %hd_job_status

hd_open_parser

defines the way to parse the magic command %hd_open

hd_pig_submit_parser

Defines the way to parse the magic command %hd_pig_submit.

hd_queue_parser

defines the way to parse the magic command %hd_queue

hd_tail_stderr_parser

defines the way to parse the magic command %hd_tail_stderr

HIVE_azure_parser

defines the way to parse the magic command %HIVE_azure

HIVE_azure_submit_parser

Defines the way to parse the magic command %HIVE_azure_submit.

jython_parser

defines the way to parse the magic command %%jython

PIG_azure_parser

defines the way to parse the magic command %%PIG_azure

runjython_parser

defines the way to parse the magic command %%runjython

Methods

method

truncated documentation

_interpret_path

interpret a path

_replace_params

replaces parameter such __PASSWORD__ by variable in the notebook environment

_run_jython

run a jython script

azureclient

returns the AzureClient object

blob_close

close the connection and remove the connection from the notebook workspace

blob_containers

returns the list of containers

blob_copy

copy a blob storage, see l-magic-path-container

blob_delete

deletes a blob, see l-magic-path-container

blob_down

download a file from the blob storage, see l-magic-path-container Example

blob_downmerge

download all files from a folder, see l-magic-path-container Example

blob_head

download a file from the blob storage and display its head, see l-magic-path-container Example

blob_ls

defines command %blob_ls, see l-magic-path-container

blob_lsl

defines command %blob_lsl (extended version of blob_lsl), see l-magic-path-container

blob_open

blob_path

checks the path used in commands blob_down, blob_up, see _interpret_path(), l-magic-path-container

blob_rm

calls blob_delete()

blob_rmr

deletes a folder, see l-magic-path-container

blob_up

upload a file to the blob storage, we assume the container is the first element of the path, see l-magic-path-container

blobcontainer

returns the Blob Storage container

blobservice

returns the BlobService object

create_client

Create a AzureClient and stores in the workspace.

get_blob_connection

returns the connection stored in the workspace

hd_job_kill

defines %hd_job_kill

hd_job_status

defines %hd_job_status

hd_open

Opens a connection to blob service.

hd_pig_submit

Defines command %hd_pig_submit.

hd_queue

defines %hd_queue

hd_tail_stderr

defines %hd_tail_stderr

hd_wasb_prefix

defines %hd_wasb_prefix, returns the prefix used to connect to the blob storage, it includes the container

HIVE_azure

defines command %%HIVE_azure

HIVE_azure_submit

Defines command %HIVE_azure_submit.

jython

Defines command %%runjython. run a jython script used for streaming in HDInsight, the function appends …

PIG_azure

defines command %%PIG_azure

runjpython

Defines command %%runjython.

Documentation

Magic command to run PIG script with Azure.

source on GitHub

class pyenbc.remote.magic_azure.MagicAzure(shell=None, **kwargs)[source]

Bases: pyquickhelper.ipythonhelper.magic_class.MagicClassWithHelpers

Defines magic commands to access blob storage and HDInsight.

When the container is not specified, it will take the default one.

Magic command %blob_open does not work

Try this:

%load_ext pyenbc

The exception tells more about what goes wrong. Usually a module is missing.

Incorrect padding

The following crypted message happens sometimes:

Error: Incorrect padding

It is usually due to an incorrect password. Some notebooks uses:

import pyquickhelper.ipythonhelper as ipy
params={"blob_storage":"hdblobstorage", "password":""}
ipy.open_html_form(params=params,title="credentials",key_save="blobservice")

blobstorage = blobservice["blob_storage"]
blobpassword = blobservice["password"]

%load_ext pyenbc
%blob_open

This code avoids the author letting password in a notebook but you can just replace everything by:

blobstorage = "<username>"
blobpassword = "****long*key*******=="

%load_ext pyenbc
%blob_open

source on GitHub

HIVE_azure(line, cell=None)[source]

defines command %%HIVE_azure

HIVE_azure

The code for magic command %HIVE_azure is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

source on GitHub

static HIVE_azure_parser()[source]

defines the way to parse the magic command %HIVE_azure

source on GitHub

HIVE_azure_submit(line)[source]

Defines command %HIVE_azure_submit.

HIVE_azure_submit

The code for magic command %HIVE_azure_submit is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.hive_submit(bs, cl.account_name, hive_file_name, dependencies, **options)

source on GitHub

static HIVE_azure_submit_parser()[source]

Defines the way to parse the magic command %HIVE_azure_submit.

source on GitHub

PIG_azure(line, cell=None)[source]

defines command %%PIG_azure

PIG_azure

The code for magic command %PIG_azure is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

source on GitHub

static PIG_azure_parser()[source]

defines the way to parse the magic command %%PIG_azure

source on GitHub

_interpret_path(line, cl, bs, empty_is_value=False)[source]

interpret a path

Parameters
  • line – line (see l-magic-path-container)

  • clAzureClient

  • bs – blob service

  • empty_is_value – if True, do not raise an exception

Returns

container, remotepath

source on GitHub

_replace_params(cell)[source]

replaces parameter such __PASSWORD__ by variable in the notebook environment

Parameters

cell – string

Returns

modified string

source on GitHub

_run_jython(cell, filename, func_name, args, true_jython=None)[source]

run a jython script

Parameters
  • cell – content of the cell

  • filename – filename used to store the content of the cell

  • func_name – function name

  • args – list of arguments to run

  • true_jython – jython (True) or this Python (False)

Returns

out, err

source on GitHub

_trait_default_generators = {}
azureclient(line)[source]

returns the AzureClient object

source on GitHub

blob_close(line)[source]

close the connection and remove the connection from the notebook workspace

blob_close

Does nothing.

source on GitHub

blob_containers(line)[source]

returns the list of containers

source on GitHub

blob_copy(line)[source]

copy a blob storage, see l-magic-path-container

blob_copy

The code for magic command %blob_copy is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.copy_blob(bs, container, dest, src)

source on GitHub

static blob_copy_parser()[source]

defines the way to parse the magic command %blob_copy

source on GitHub

blob_delete(line)[source]

deletes a blob, see l-magic-path-container

blob_delete

The code for magic command %blob_delete is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.delete_blob(bs, container, remotepath)

source on GitHub

static blob_delete_parser()[source]

defines the way to parse the magic command %blob_delete

source on GitHub

blob_down(line)[source]

download a file from the blob storage, see l-magic-path-container

Example:

%blob_down remotepath localfile

the command does not allow spaces in file names

blob_down

The code for magic command %blob_down is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.download(bs, container, remotepath, localfile)

source on GitHub

static blob_down_parser()[source]

Defines the way to parse the magic command %blob_down.

source on GitHub

blob_downmerge(line)[source]

download all files from a folder, see l-magic-path-container

Example:

%blob_downmerge remotepath localfile

the command does not allow spaces in file names

blob_downmerge

The code for magic command %blob_downmerge is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.download_merge(bs, container, remotepath, localfile)

New in version 1.1.

source on GitHub

static blob_downmerge_parser()[source]

defines the way to parse the magic command %blob_downmerge

source on GitHub

blob_head(line)[source]

download a file from the blob storage and display its head, see l-magic-path-container

Example:

%blob_head remotepath

the command does not allow spaces in file names

blob_head

The code for magic command %blob_head is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.df_head(bs, container, remotepath, localfile)

source on GitHub

static blob_head_parser()[source]

defines the way to parse the magic command %blob_head

source on GitHub

blob_ls(line)[source]

defines command %blob_ls, see l-magic-path-container

blob_ls

The code for magic command %blob_ls is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.ls(bs, container, remotepath)

source on GitHub

static blob_ls_parser()[source]

defines the way to parse the magic command %blob_ls

source on GitHub

blob_lsl(line)[source]

defines command %blob_lsl (extended version of blob_lsl), see l-magic-path-container

blob_lsl

The code for magic command %blob_lsl is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.ls(bs, container, remotepath, add_metadata=True)

source on GitHub

static blob_lsl_parser()[source]

defines the way to parse the magic command %blob_lsl

source on GitHub

blob_open(line)[source]

blob_open

Opens a connection to blob service. It returns objects AzureClient and BlobService.

The code for magic command %blob_open is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()

Changed in version 1.1.

source on GitHub

static blob_open_parser()[source]

defines the way to parse the magic command %blob_open

source on GitHub

blob_path(line)[source]

checks the path used in commands blob_down, blob_up, see _interpret_path(), l-magic-path-container

blob_path

The code for magic command %blob_path is equivalent to:

if line.startswith("/"):
    container = account_name
    remotepath = remotepath.lstrip("/")
else:
    spl = line.split("/")
    container = spl[0]
    remotepath = None if len(spl) == 1 else "/".join(spl[1:])

result = container, remotepath

source on GitHub

static blob_path_parser()[source]

defines the magic command %blob_path, checks the path used in commands blob_down, blob_up

source on GitHub

blob_rm(line)[source]

calls blob_delete

New in version 1.1.

source on GitHub

blob_rmr(line)[source]

deletes a folder, see l-magic-path-container

blob_rmr

The code for magic command %blob_rmr is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.delete_folder(bs, container, remotepath)

source on GitHub

static blob_rmr_parser()[source]

defines the way to parse the magic command %blob_rmr

source on GitHub

blob_up(line)[source]

upload a file to the blob storage, we assume the container is the first element of the path, see l-magic-path-container

Example:

%blob_up localfile remotepath

the command does not allow spaces in files

blob_up

The code for magic command %blob_up is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.upload(bs, container, remotepath, localfile)

source on GitHub

static blob_up_parser()[source]

Defines the way to parse the magic command %blob_up.

source on GitHub

blobcontainer(line)[source]

returns the Blob Storage container

source on GitHub

blobservice(line)[source]

returns the BlobService object

source on GitHub

create_client(account_name, account_key, hadoop_server=None, hadoop_password=None, username=None)[source]

Create a AzureClient and stores in the workspace.

Parameters
  • account_name – login

  • account_key – password

  • hadoop_server – hadoop server

  • hadoop_password – hadoop password

  • username – username

Returns

instance of AzureClient

source on GitHub

get_blob_connection()[source]

returns the connection stored in the workspace

source on GitHub

hd_job_kill(line)[source]

defines %hd_job_kill

hd_job_kill

The code for magic command %hd_job_kill is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_kill(jobid)

source on GitHub

static hd_job_kill_parser()[source]

defines the way to parse the magic command %hd_job_kill

source on GitHub

hd_job_status(line)[source]

defines %hd_job_status

hd_job_status

The code for magic command %hd_job_status is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_status(jobid)

source on GitHub

static hd_job_status_parser()[source]

defines the way to parse the magic command %hd_job_status

source on GitHub

hd_open(line)[source]

Opens a connection to blob service.

hd_open

Opens a connection to blob service. It returns objects AzureClient and BlobService.

The code for magic command %hd_open is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()

source on GitHub

static hd_open_parser()[source]

defines the way to parse the magic command %hd_open

source on GitHub

hd_pig_submit(line)[source]

Defines command %hd_pig_submit.

hd_pig_submit

The code for magic command %hd_pig_submit is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.pig_submit(bs, cl.account_name, pig_file_name, dependencies, **options)

source on GitHub

static hd_pig_submit_parser()[source]

Defines the way to parse the magic command %hd_pig_submit.

source on GitHub

hd_queue(line)[source]

defines %hd_queue

hd_queue

The code for magic command %hd_queue is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_queue(showall=showall)

source on GitHub

static hd_queue_parser()[source]

defines the way to parse the magic command %hd_queue

source on GitHub

hd_tail_stderr(line)[source]

defines %hd_tail_stderr

Warning

This function gets the status of the job to get the script name. But the rediction uses the script name and not the job id. As a consequence, if the same script name is run multiple times, the redirection will contain the output of multiples jobs.

hd_tail_stderr

The code for magic command %hd_tail_stderr is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.standard_outputs(job_id, bs, cl.account_name, ".")

source on GitHub

static hd_tail_stderr_parser()[source]

defines the way to parse the magic command %hd_tail_stderr

source on GitHub

hd_wasb_prefix(line)[source]

defines %hd_wasb_prefix, returns the prefix used to connect to the blob storage, it includes the container name

source on GitHub

jython(line, cell=None)[source]

Defines command %%runjython.

run a jython script used for streaming in HDInsight, the function appends fake decorator a timeout is set up at 10s

The magic function create another file included the decoration. It runs the script with Jython (see the default version)

See In a python script how can I ignore Apache Pig’s Python Decorators for standalone unit testing.

New in version 1.1.

source on GitHub

static jython_parser()[source]

defines the way to parse the magic command %%jython

source on GitHub

runjpython(line, cell=None)[source]

Defines command %%runjython.

runjpython

Run a jython script used for streaming in HDInsight, the function appends fake decorator a timeout is set up at 10s

The magic function create another file included the decoration. It runs the script with this version of Python.

See In a python script how can I ignore Apache Pig’s Python Decorators for standalone unit testing

See _run_jython to see the code.

New in version 1.1.

source on GitHub

static runjython_parser()[source]

defines the way to parse the magic command %%runjython

source on GitHub

pyenbc.remote.magic_azure.register_azure_magics(ip=None)[source]

register magics function, can be called from a notebook

Parameters

ip – from get_ipython()

source on GitHub