Magic commands

Azure

  1. HIVE_azure

  2. HIVE_azure_submit

  3. PIG_azure

  4. blob_close

  5. blob_copy

  6. blob_delete

  7. blob_down

  8. blob_downmerge

  9. blob_head

  10. blob_ls

  11. blob_lsl

  12. blob_open

  13. blob_path

  14. blob_rmr

  15. blob_up

  16. hd_job_kill

  17. hd_job_status

  18. hd_open

  19. hd_pig_submit

  20. hd_queue

  21. hd_tail_stderr

  22. runjpython

HIVE_azure

The code for magic command %HIVE_azure is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.HIVE_azure, line 3)

HIVE_azure_submit

The code for magic command %HIVE_azure_submit is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.hive_submit(bs, cl.account_name, hive_file_name, dependencies, **options)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.HIVE_azure_submit, line 3)

PIG_azure

The code for magic command %PIG_azure is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.PIG_azure, line 3)

blob_close

Does nothing.

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_close, line 4)

blob_copy

The code for magic command %blob_copy is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.copy_blob(bs, container, dest, src)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_copy, line 4)

blob_delete

The code for magic command %blob_delete is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.delete_blob(bs, container, remotepath)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_delete, line 4)

blob_down

The code for magic command %blob_down is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.download(bs, container, remotepath, localfile)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_down, line 10)

blob_downmerge

The code for magic command %blob_downmerge is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.download_merge(bs, container, remotepath, localfile)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_downmerge, line 10)

blob_head

The code for magic command %blob_head is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.df_head(bs, container, remotepath, localfile)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_head, line 10)

blob_ls

The code for magic command %blob_ls is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.ls(bs, container, remotepath)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_ls, line 3)

blob_lsl

The code for magic command %blob_lsl is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.ls(bs, container, remotepath, add_metadata=True)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_lsl, line 4)

blob_open

Opens a connection to blob service. It returns objects AzureClient and BlobService.

The code for magic command %blob_open is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_open, line 1)

blob_path

The code for magic command %blob_path is equivalent to:

if line.startswith("/"):
    container = account_name
    remotepath = remotepath.lstrip("/")
else:
    spl = line.split("/")
    container = spl[0]
    remotepath = None if len(spl) == 1 else "/".join(spl[1:])

result = container, remotepath

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_path, line 4)

blob_rmr

The code for magic command %blob_rmr is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.delete_folder(bs, container, remotepath)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_rmr, line 4)

blob_up

The code for magic command %blob_up is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.upload(bs, container, remotepath, localfile)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_up, line 11)

hd_job_kill

The code for magic command %hd_job_kill is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_kill(jobid)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_job_kill, line 3)

hd_job_status

The code for magic command %hd_job_status is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_status(jobid)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_job_status, line 3)

hd_open

Opens a connection to blob service. It returns objects AzureClient and BlobService.

The code for magic command %hd_open is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_open, line 3)

hd_pig_submit

The code for magic command %hd_pig_submit is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.pig_submit(bs, cl.account_name, pig_file_name, dependencies, **options)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_pig_submit, line 3)

hd_queue

The code for magic command %hd_queue is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_queue(showall=showall)

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_queue, line 3)

hd_tail_stderr

The code for magic command %hd_tail_stderr is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.standard_outputs(job_id, bs, cl.account_name, ".")

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_tail_stderr, line 9)

runjpython

Run a jython script used for streaming in HDInsight, the function appends fake decorator a timeout is set up at 10s

The magic function create another file included the decoration. It runs the script with this version of Python.

See In a python script how can I ignore Apache Pig’s Python Decorators for standalone unit testing

See _run_jython to see the code.

(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.runjpython, line 3)

Hadoop

  1. HIVE

  2. PIG

  3. dfs_ls

  4. dfs_mkdir

  5. dfs_rm

  6. hive_submit

  7. open_remote_shell

  8. pig_submit

  9. remote_close

  10. remote_cmd

  11. remote_down

  12. remote_down_cluster

  13. remote_ls

  14. remote_open

  15. remote_py

  16. remote_up

  17. remote_up_cluster

HIVE

The code for magic command %HIVE is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.HIVE, line 3)

PIG

The code for magic command %PIG is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.PIG, line 3)

dfs_ls

The code for magic command %dfs_ls is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.dfs_ls(args.path)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.dfs_ls, line 7)

dfs_mkdir

The code for magic command %dfs_mkdir is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.dfs_mkdir(path)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.dfs_mkdir, line 7)

dfs_rm

The code for magic command %dfs_rm is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.dfs_rm(path, recursive=recursive)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.dfs_rm, line 7)

hive_submit

The code for magic command %hive_submit is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
out, err = ssh.hive_submit(
    pig, redirection=redirection, local=local)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.hive_submit, line 3)

open_remote_shell

The code for magic command %open_remote_shell is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.open_session(out_format=format)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.open_remote_shell, line 3)

pig_submit

The code for magic command %pig_submit is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
out, err = ssh.pig_submit(
    pig, dependencies=dependencies, redirection=redirection, local=local, stop_on_failure=stop_on_failure)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.pig_submit, line 3)

remote_close

The code for magic command %remote_close is equivalent to:

ssh = ASSHClient(server, username, password)
# ... ssh.connect()
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_close, line 4)

remote_cmd

The code for magic command %remote_cmd is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
out, err = ssh.execute_command(
    line, no_exception=True, fill_stdin=cell)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_cmd, line 15)

remote_down

The code for magic command %remote_down is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.download(remotepath, localfile)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_down, line 9)

remote_down_cluster

The code for magic command %remote_down_cluster is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.download_cluster(remotepath, localfile, merge=merge)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_down_cluster, line 9)

remote_ls

The code for magic command %remote_ls is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.ls(path)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_ls, line 7)

remote_open

The code for magic command %remote_open is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_open, line 4)

remote_py

The code for magic command %remote_py is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.upload(filename, dest)
args = " ".join('"{}"'.format(_)
                for _ in args.args) if args.args is not None else ""
out, err = ssh.execute_command(exe + " " + dest + " " + args, no_exception=True)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_py, line 3)

remote_up

The code for magic command %remote_up is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.upload(localfile, remotepath)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_up, line 9)

remote_up_cluster

The code for magic command %remote_up_cluster is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.upload_cluster(localfile, remotepath)
ssh.close()

(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_up_cluster, line 9)