module remote.magic_remote_ssh

Inheritance diagram of pyenbc.remote.magic_remote_ssh

Short summary

module pyenbc.remote.magic_remote_ssh

Magic command to communicate with an Hadoop cluster (Linux).

source on GitHub

Classes

class

truncated documentation

MagicRemoteSSH

Defines commands to access a remote machine (bridge) through SSH, for the time being, all the command do not accept …

Functions

function

truncated documentation

register_magics_ssh

register magics function, can be called from a notebook

Properties

property

truncated documentation

Context

return the context or None

cross_validation_lock

A contextmanager for running a block with our cross validation lock set to True. At the end of the block, …

Static Methods

staticmethod

truncated documentation

dfs_ls_parser

defines the way to parse the magic command %dfs_ls

dfs_mkdir_parser

defines the way to parse the magic command %dfs_mkdir

dfs_rm_parser

defines the way to parse the magic command %dfs_rm

HIVE_parser

defines the way to parse the magic command %%HIVE

hive_submit_parser

defines the way to parse the magic command %hive_submit

job_syntax_parser

defines the way to parse the magic command %job_syntax

open_remote_shell_parser

defines the way to parse the magic command %open_remote_shell

PIG_parser

defines the way to parse the magic command %%PIG

pig_submit_parser

defines the way to parse the magic command %pig_submit

remote_down_cluster_parser

defines the way to parse the magic command %remote_down_cluster

remote_down_parser

defines the way to parse the magic command %remote_down

remote_ls_parser

defines the way to parse the magic command %remote_ls

remote_open_parser

defines the way to parse the magic command %remote_open

remote_py_parser

defines the way to parse the magic command %remote_py

remote_up_cluster_parser

defines the way to parse the magic command %remote_up_cluster

remote_up_parser

defines the way to parse the magic command %remote_up

Methods

method

truncated documentation

_replace_params

replaces parameter such __PASSWORD__ by variable in the notebook environment

close_remote_shell

Defines %close_remote_shell

dfs_ls

returns the content of a folder on the cluster as a dataframe Example

dfs_mkdir

Creates a folder on the cluster. Example

dfs_rm

remove a file on the cluster Example

get_connection

returns the connection stored in the workspace

HIVE

defines command %%HIVE

hive_submit

defines command %hive_submit

job_syntax

defines command %job_syntax

open_remote_shell

Defines %open_remote_shell

PIG

defines command %%PIG

pig_submit

defines command %pig_submit

remote_close

close a SSH connection and store the connection into the notebook workspace

remote_cmd

run a command on the remote machine Example

remote_cmd_text

run a command on the remote machine and returns raw text (not HTML) Example

remote_down

download a file from the remote machine, Example

remote_down_cluster

download a file from the cluster to the local machine through the bridge Example

remote_ls

returns the content of a folder on the remote machine as a dataframe Example

remote_open

open a SSH connection and store the connection into the notebook workspace

remote_py

defines command %remote_py

remote_up

upload a file to the remote machine, Example

remote_up_cluster

upload a file to the remote machine and then to the remote cluster, Example

shell_remote

Defines %shell_remote and %%shell_remote

shell_remote_text

Defines %shell_remote_text and %%shell_remote_text

Documentation

Magic command to communicate with an Hadoop cluster (Linux).

source on GitHub

class pyenbc.remote.magic_remote_ssh.MagicRemoteSSH(shell=None, **kwargs)[source]

Bases: pyquickhelper.ipythonhelper.magic_class.MagicClassWithHelpers

Defines commands to access a remote machine (bridge) through SSH, for the time being, all the command do not accept another parameters such as a SSH client which means only one connection can be opened at the same time.

The notebooks Python Hadoop Pig and Communication with a remote Linux machine through SSH show how these commands can be used.

source on GitHub

HIVE(line, cell=None)[source]

defines command %%HIVE

HIVE

The code for magic command %HIVE is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

source on GitHub

static HIVE_parser()[source]

defines the way to parse the magic command %%HIVE

source on GitHub

PIG(line, cell=None)[source]

defines command %%PIG

PIG

The code for magic command %PIG is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

source on GitHub

static PIG_parser()[source]

defines the way to parse the magic command %%PIG

source on GitHub

_replace_params(cell)[source]

replaces parameter such __PASSWORD__ by variable in the notebook environment

Parameters

cell – string

Returns

modified string

source on GitHub

_trait_default_generators = {}
close_remote_shell(line)[source]

Defines %close_remote_shell

source on GitHub

dfs_ls(line)[source]

returns the content of a folder on the cluster as a dataframe

Example:

%dfs_ls .

dfs_ls

The code for magic command %dfs_ls is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.dfs_ls(args.path)
ssh.close()

New in version 1.1.

source on GitHub

static dfs_ls_parser()[source]

defines the way to parse the magic command %dfs_ls

source on GitHub

dfs_mkdir(line)[source]

Creates a folder on the cluster.

Example:

%dfs_mkdir afolder

dfs_mkdir

The code for magic command %dfs_mkdir is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.dfs_mkdir(path)
ssh.close()

New in version 1.1.

source on GitHub

static dfs_mkdir_parser()[source]

defines the way to parse the magic command %dfs_mkdir

source on GitHub

dfs_rm(line)[source]

remove a file on the cluster

Example:

%dfs_rm .

dfs_rm

The code for magic command %dfs_rm is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.dfs_rm(path, recursive=recursive)
ssh.close()

New in version 1.1.

source on GitHub

static dfs_rm_parser()[source]

defines the way to parse the magic command %dfs_rm

source on GitHub

get_connection()[source]

returns the connection stored in the workspace

source on GitHub

hive_submit(line)[source]

defines command %hive_submit

hive_submit

The code for magic command %hive_submit is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
out, err = ssh.hive_submit(
    pig, redirection=redirection, local=local)
ssh.close()

source on GitHub

static hive_submit_parser()[source]

defines the way to parse the magic command %hive_submit

source on GitHub

job_syntax(line)[source]

defines command %job_syntax

source on GitHub

static job_syntax_parser()[source]

defines the way to parse the magic command %job_syntax

source on GitHub

open_remote_shell(line)[source]

Defines %open_remote_shell

open_remote_shell

The code for magic command %open_remote_shell is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.open_session(out_format=format)
ssh.close()

source on GitHub

static open_remote_shell_parser()[source]

defines the way to parse the magic command %open_remote_shell

source on GitHub

pig_submit(line)[source]

defines command %pig_submit

pig_submit

The code for magic command %pig_submit is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
out, err = ssh.pig_submit(
    pig, dependencies=dependencies, redirection=redirection, local=local, stop_on_failure=stop_on_failure)
ssh.close()

source on GitHub

static pig_submit_parser()[source]

defines the way to parse the magic command %pig_submit

source on GitHub

remote_close(line)[source]

close a SSH connection and store the connection into the notebook workspace

remote_close

The code for magic command %remote_close is equivalent to:

ssh = ASSHClient(server, username, password)
# ... ssh.connect()
ssh.close()

source on GitHub

remote_cmd(line, cell=None)[source]

run a command on the remote machine

Example:

%remote_cmd ls

Or:

%%remote_cmd  <something>
anything going to stdin

In the second case, if __PASSWORD__ is found, it will be replaced by the password stored in workspace.

remote_cmd

The code for magic command %remote_cmd is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
out, err = ssh.execute_command(
    line, no_exception=True, fill_stdin=cell)
ssh.close()

source on GitHub

remote_cmd_text(line, cell=None)[source]

run a command on the remote machine and returns raw text (not HTML)

Example:

%remote_cmd_text ls

Or:

%%remote_cmd_text  <something>
anything going to stdin

In the second case, if __PASSWORD__ is found, it will be replaced by the password stored in workspace.

source on GitHub

remote_down(line)[source]

download a file from the remote machine,

Example:

%remote_down remotepath localfile

the command does not allow spaces in files

remote_down

The code for magic command %remote_down is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.download(remotepath, localfile)
ssh.close()

source on GitHub

remote_down_cluster(line)[source]

download a file from the cluster to the local machine through the bridge

Example:

%remote_down_cluster remotepath localfile

the command does not allow spaces in files

remote_down_cluster

The code for magic command %remote_down_cluster is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.download_cluster(remotepath, localfile, merge=merge)
ssh.close()

New in version 1.1.

source on GitHub

static remote_down_cluster_parser()[source]

defines the way to parse the magic command %remote_down_cluster

source on GitHub

static remote_down_parser()[source]

defines the way to parse the magic command %remote_down

source on GitHub

remote_ls(line)[source]

returns the content of a folder on the remote machine as a dataframe

Example:

%remote_ls .

remote_ls

The code for magic command %remote_ls is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
df = ssh.ls(path)
ssh.close()

New in version 1.1.

source on GitHub

static remote_ls_parser()[source]

defines the way to parse the magic command %remote_ls

source on GitHub

remote_open(line)[source]

open a SSH connection and store the connection into the notebook workspace

remote_open

The code for magic command %remote_open is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()

source on GitHub

static remote_open_parser()[source]

defines the way to parse the magic command %remote_open

source on GitHub

remote_py(line)[source]

defines command %remote_py

remote_py

The code for magic command %remote_py is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.upload(filename, dest)
args = " ".join('"{}"'.format(_)
                for _ in args.args) if args.args is not None else ""
out, err = ssh.execute_command(exe + " " + dest + " " + args, no_exception=True)
ssh.close()

source on GitHub

static remote_py_parser()[source]

defines the way to parse the magic command %remote_py

source on GitHub

remote_up(line)[source]

upload a file to the remote machine,

Example:

%remote_up localfile remotepath

the command does not allow spaces in files

remote_up

The code for magic command %remote_up is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.upload(localfile, remotepath)
ssh.close()

source on GitHub

remote_up_cluster(line)[source]

upload a file to the remote machine and then to the remote cluster,

Example:

%remote_up_cluster localfile remotepath

the command does not allow spaces in files

remote_up_cluster

The code for magic command %remote_up_cluster is equivalent to:

ssh = ASSHClient(server, username, password)
ssh.connect()
ssh.upload_cluster(localfile, remotepath)
ssh.close()

New in version 1.1.

source on GitHub

static remote_up_cluster_parser()[source]

defines the way to parse the magic command %remote_up_cluster

source on GitHub

static remote_up_parser()[source]

defines the way to parse the magic command %remote_up

source on GitHub

shell_remote(line, cell=None)[source]

Defines %shell_remote and %%shell_remote

source on GitHub

shell_remote_text(line, cell=None)[source]

Defines %shell_remote_text and %%shell_remote_text

source on GitHub

pyenbc.remote.magic_remote_ssh.register_magics_ssh(ip=None)[source]

register magics function, can be called from a notebook

Parameters

ip – from get_ipython()

source on GitHub