Magic commands¶
Azure¶
HIVE_azure
The code for magic command %HIVE_azure
is equivalent to:
with open(filename, "w", encoding="utf8") as f:
f.write(script)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.HIVE_azure, line 3)
HIVE_azure_submit
The code for magic command %HIVE_azure_submit
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.hive_submit(bs, cl.account_name, hive_file_name, dependencies, **options)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.HIVE_azure_submit, line 3)
PIG_azure
The code for magic command %PIG_azure
is equivalent to:
with open(filename, "w", encoding="utf8") as f:
f.write(script)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.PIG_azure, line 3)
blob_close
Does nothing.
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_close, line 4)
blob_copy
The code for magic command %blob_copy
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.copy_blob(bs, container, dest, src)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_copy, line 4)
blob_delete
The code for magic command %blob_delete
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.delete_blob(bs, container, remotepath)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_delete, line 4)
blob_down
The code for magic command %blob_down
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.download(bs, container, remotepath, localfile)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_down, line 10)
blob_downmerge
The code for magic command %blob_downmerge
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.download_merge(bs, container, remotepath, localfile)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_downmerge, line 10)
blob_head
The code for magic command %blob_head
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.df_head(bs, container, remotepath, localfile)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_head, line 10)
blob_ls
The code for magic command %blob_ls
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.ls(bs, container, remotepath)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_ls, line 3)
blob_lsl
The code for magic command %blob_lsl
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.ls(bs, container, remotepath, add_metadata=True)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_lsl, line 4)
blob_open
Opens a connection to blob service.
It returns objects AzureClient
and
BlobService.
The code for magic command %blob_open
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_open, line 1)
blob_path
The code for magic command %blob_path
is equivalent to:
if line.startswith("/"):
container = account_name
remotepath = remotepath.lstrip("/")
else:
spl = line.split("/")
container = spl[0]
remotepath = None if len(spl) == 1 else "/".join(spl[1:])
result = container, remotepath
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_path, line 4)
blob_rmr
The code for magic command %blob_rmr
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.delete_folder(bs, container, remotepath)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_rmr, line 4)
blob_up
The code for magic command %blob_up
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.upload(bs, container, remotepath, localfile)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.blob_up, line 11)
hd_job_kill
The code for magic command %hd_job_kill
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_kill(jobid)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_job_kill, line 3)
hd_job_status
The code for magic command %hd_job_status
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_status(jobid)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_job_status, line 3)
hd_open
Opens a connection to blob service.
It returns objects AzureClient
and
BlobService.
The code for magic command %hd_open
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_open, line 3)
hd_pig_submit
The code for magic command %hd_pig_submit
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.pig_submit(bs, cl.account_name, pig_file_name, dependencies, **options)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_pig_submit, line 3)
hd_queue
The code for magic command %hd_queue
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_queue(showall=showall)
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_queue, line 3)
hd_tail_stderr
The code for magic command %hd_tail_stderr
is equivalent to:
from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.standard_outputs(job_id, bs, cl.account_name, ".")
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.hd_tail_stderr, line 9)
runjpython
Run a jython script used for streaming in HDInsight, the function appends fake decorator a timeout is set up at 10s
The magic function create another file included the decoration. It runs the script with this version of Python.
See In a python script how can I ignore Apache Pig’s Python Decorators for standalone unit testing
See _run_jython
to see the code.
(original entry : magic_azure.py:docstring of pyenbc.remote.magic_azure.MagicAzure.runjpython, line 3)
Hadoop¶
HIVE
The code for magic command %HIVE
is equivalent to:
with open(filename, "w", encoding="utf8") as f:
f.write(script)
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.HIVE, line 3)
PIG
The code for magic command %PIG
is equivalent to:
with open(filename, "w", encoding="utf8") as f:
f.write(script)
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.PIG, line 3)
dfs_ls
The code for magic command %dfs_ls
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.dfs_ls(args.path)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.dfs_ls, line 7)
dfs_mkdir
The code for magic command %dfs_mkdir
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.dfs_mkdir(path)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.dfs_mkdir, line 7)
dfs_rm
The code for magic command %dfs_rm
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.dfs_rm(path, recursive=recursive)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.dfs_rm, line 7)
hive_submit
The code for magic command %hive_submit
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
out, err = ssh.hive_submit(
pig, redirection=redirection, local=local)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.hive_submit, line 3)
open_remote_shell
The code for magic command %open_remote_shell
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.open_session(out_format=format)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.open_remote_shell, line 3)
pig_submit
The code for magic command %pig_submit
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
out, err = ssh.pig_submit(
pig, dependencies=dependencies, redirection=redirection, local=local, stop_on_failure=stop_on_failure)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.pig_submit, line 3)
remote_close
The code for magic command %remote_close
is equivalent to:
ssh = ASSHClient(server, username, password)
# ... ssh.connect()
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_close, line 4)
remote_cmd
The code for magic command %remote_cmd
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
out, err = ssh.execute_command(
line, no_exception=True, fill_stdin=cell)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_cmd, line 15)
remote_down
The code for magic command %remote_down
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.download(remotepath, localfile)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_down, line 9)
remote_down_cluster
The code for magic command %remote_down_cluster
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.download_cluster(remotepath, localfile, merge=merge)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_down_cluster, line 9)
remote_ls
The code for magic command %remote_ls
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.ls(path)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_ls, line 7)
remote_open
The code for magic command %remote_open
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_open, line 4)
remote_py
The code for magic command %remote_py
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.upload(filename, dest)
args = " ".join('"{}"'.format(_)
for _ in args.args) if args.args is not None else ""
out, err = ssh.execute_command(exe + " " + dest + " " + args, no_exception=True)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_py, line 3)
remote_up
The code for magic command %remote_up
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.upload(localfile, remotepath)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_up, line 9)
remote_up_cluster
The code for magic command %remote_up_cluster
is equivalent to:
ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.upload_cluster(localfile, remotepath)
ssh.close()
(original entry : magic_remote_ssh.py:docstring of pyenbc.remote.magic_remote_ssh.MagicRemoteSSH.remote_up_cluster, line 9)