Methods

Summary

method class parent truncated documentation
HIVE MagicRemoteSSH defines command %%HIVE
HIVE_azure MagicAzure defines command %%HIVE_azure
HIVE_azure_submit MagicAzure Defines command %HIVE_azure_submit.
PIG MagicRemoteSSH defines command %%PIG
PIG_azure MagicAzure defines command %%PIG_azure
__init__ AzureClient constructor
__init__ AzureDrive constructor
__init__ AzureException store more information than a regular exception
__init__ AzureTransferAPI  
__init__ CloudTransfer constructor
__init__ ASSHClient constructor
__str__ AzureClient usual
__str__ AzureException usual
__str__ ASSHClient usual
_interpret_path AzureClient replace variavble such as $PSEUDO, $USERNAME
_interpret_path MagicAzure interpret a path
_replace_params MagicAzure replaces parameter such __PASSWORD__ by variable in the notebook environment
_replace_params MagicRemoteSSH replaces parameter such __PASSWORD__ by variable in the notebook environment
_run_jython MagicAzure run a jython script
azureclient MagicAzure returns the AzureClient object
blob_close MagicAzure close the connection and remove the connection from the notebook workspace
blob_containers MagicAzure returns the list of containers
blob_copy MagicAzure copy a blob storage, see l-magic-path-container
blob_delete MagicAzure deletes a blob, see l-magic-path-container
blob_down MagicAzure download a file from the blob storage, see l-magic-path-container Example
blob_downmerge MagicAzure download all files from a folder, see l-magic-path-container Example
blob_head MagicAzure download a file from the blob storage and display its head, see l-magic-path-container Example
blob_ls MagicAzure defines command %blob_ls, see l-magic-path-container
blob_lsl MagicAzure defines command %blob_lsl (extended version of blob_lsl), see l-magic-path-container
blob_open MagicAzure  
blob_path MagicAzure checks the path used in commands blob_down, blob_up, see _interpret_path(), l-magic-path-container
blob_rm MagicAzure calls blob_delete()
blob_rmr MagicAzure deletes a folder, see l-magic-path-container
blob_up MagicAzure upload a file to the blob storage, we assume the container is the first element of the path, see l-magic-path-container
blobcontainer MagicAzure returns the Blob Storage container
blobservice MagicAzure returns the BlobService object
close AzureDrive close the connection
close CloudTransfer close the connection
close ASSHClient close the connection
close_remote_shell MagicRemoteSSH Defines %close_remote_shell
close_session ASSHClient close a session
connect AzureDrive connect
connect CloudTransfer connect
connect ASSHClient connect
copy_blob AzureClient copy a blob
create_client MagicAzure Create a AzureClient and stores in the workspace.
delete_blob AzureClient delete a blob
delete_folder AzureClient delete a folder and its content
df_head AzureClient Downloads the beginning of a stream and displays as a DataFrame.
dfs_exists ASSHClient tells if a file exists on the cluster
dfs_ls MagicRemoteSSH returns the content of a folder on the cluster as a dataframe Example
dfs_ls ASSHClient return the content of a folder on the cluster as a DataFrame
dfs_mkdir MagicRemoteSSH Creates a folder on the cluster. Example
dfs_mkdir ASSHClient creates a directory on the cluster
dfs_rm MagicRemoteSSH remove a file on the cluster Example
dfs_rm ASSHClient removes a file on the cluster
download AzureClient Downloads data from a blob storage to a file. No more than 64Mb can be downloaded at the same, it needs to be split …
download ASSHClient download a file from the remote machine (not on the cluster)
download_cluster ASSHClient download a file directly from the cluster to the local machine
download_data AzureClient Downloads data from a blob storage and return bytes. No more than 64Mb can be downloaded at the same, it needs …
download_data AzureDrive download binary data
download_data CloudTransfer download binary data
download_merge AzureClient Downloads all files from a folder in a blob storage to a single local file. Files will be merged. No more …
execute_command ASSHClient execute a command line, it raises an error if there is an error
exists AzureClient test the existence of a path on the blob storage
exists ASSHClient tells if a file exists on the bridge
get_blob_connection MagicAzure returns the connection stored in the workspace
get_connection MagicRemoteSSH returns the connection stored in the workspace
get_status AzureClient return the status of the webHCatUrl server
get_version AzureClient return the status of the WebHCat version
hd_job_kill MagicAzure defines %hd_job_kill
hd_job_status MagicAzure defines %hd_job_status
hd_open MagicAzure Opens a connection to blob service.
hd_pig_submit MagicAzure Defines command %hd_pig_submit.
hd_queue MagicAzure defines %hd_queue
hd_tail_stderr MagicAzure defines %hd_tail_stderr
hd_wasb_prefix MagicAzure defines %hd_wasb_prefix, returns the prefix used to connect to the blob storage, it includes the container
hive_submit AzureClient Submits a :epkg:`HIVE` job, the function uploads it to the cluster as well as the dependencies. The code …
hive_submit MagicRemoteSSH defines command %hive_submit
hive_submit ASSHClient submits a PIG script, it first upload the script to the default folder and submit it
job_kill AzureClient kills a job see Delete Job — DELETE queue/:jobid
job_queue AzureClient returns the list of jobs It uses the API Job Information — GET queue/:jobid. …
job_status AzureClient return the status of a job see List Versions — GET version
job_syntax MagicRemoteSSH defines command %job_syntax
jython MagicAzure Defines command %%runjython. run a jython script used for streaming in HDInsight, the function appends …
ls AzureClient return the content of a blob storage
ls ASSHClient return the content of a folder on the bridge as a DataFrame
open_blob_service AzureClient open a blob service
open_remote_shell MagicRemoteSSH Defines %open_remote_shell
open_session ASSHClient Opens a session with method invoke_shell. …
pig_submit AzureClient Submits a :epkg:`PIG` job, the function uploads it to the cluster as well as the dependencies. The code …
pig_submit MagicRemoteSSH defines command %pig_submit
pig_submit ASSHClient Submits a :epkg:`PIG` script, it first upload the script to the default folder and submits it.
remote_close MagicRemoteSSH close a SSH connection and store the connection into the notebook workspace
remote_cmd MagicRemoteSSH run a command on the remote machine Example
remote_cmd_text MagicRemoteSSH run a command on the remote machine and returns raw text (not HTML) Example
remote_down MagicRemoteSSH download a file from the remote machine, Example
remote_down_cluster MagicRemoteSSH download a file from the cluster to the local machine through the bridge Example
remote_ls MagicRemoteSSH returns the content of a folder on the remote machine as a dataframe Example
remote_open MagicRemoteSSH open a SSH connection and store the connection into the notebook workspace
remote_py MagicRemoteSSH defines command %remote_py
remote_up MagicRemoteSSH upload a file to the remote machine, Example
remote_up_cluster MagicRemoteSSH upload a file to the remote machine and then to the remote cluster, Example
retrieve AzureTransferAPI retrieve data from path
runjpython MagicAzure Defines command %%runjython.
send_recv_session ASSHClient Send something through a session, the function is supposed to return when the execute of the given command is done, …
shell_remote MagicRemoteSSH Defines %shell_remote and %%shell_remote
shell_remote_text MagicRemoteSSH Defines %shell_remote_text and %%shell_remote_text
standard_outputs AzureClient returns the standard output and error for a specific job id
transfer AzureTransferAPI we assume a data holds in memory, tansfer data to path
upload AzureClient Uploads data from a file to a blob storage. No more than 64Mb can be uploaded at the same, it needs to be split …
upload ASSHClient upload a file to the remote machine (not on the cluster)
upload_cluster ASSHClient the function directly uploads the file to the cluster, it first goes to the bridge, uploads it to the cluster and …
upload_data AzureClient Uploads data (bytes) to a blob storage. No more than 64Mb can be uploaded at the same, it needs to be split into …
upload_data AzureDrive upload binary data
upload_data CloudTransfer upload binary data
url_blob AzureClient returns an url for a blob file name
url_webHCatUrl AzureClient returns an url to the cluster
wait_job AzureClient wait until a job has run or failed
wasb_prefix AzureClient when using an instruction LOAD in a PIG script, file blob name must be reference using a wasb syntax. …
wasb_to_file AzureClient return something like wasb://demo@myblobstorage.blob...