Communication with a remote Linux machine through SSH

Links: notebook, html, PDF, python, slides, GitHub

In this notebook, we show how to communicate with a remote machine. This machine was set up using Ubuntu 12 (and with Azure : Create a Virtual Machine Running Linux). The following code is used to get the credentials without printing them on the notebook.

%load_ext pyensae
%load_ext pyenbc
import pyquickhelper.ipythonhelper as ipy
params={"server":"", "username":"", "password":""}
ipy.open_html_form(params=params,title="credentials",key_save="ssh_remote")
credentials
password
server
username
password = ssh_remote["password"]
server = ssh_remote["server"]
username = ssh_remote["username"]

We are going to use the following magic commands MagicRemote:

from pyense.remote import ASSHClient
help(ASSHClient)
Help on class ASSHClient in module pyenbc.remote.ssh_remote_connection:
class ASSHClient(builtins.object)
 |  A simple class to access to remote machine through SSH.
 |  It requires modules
 |  paramiko,
 |  pycrypto,
 |  ecdsa.
 |
 |  This class is used in magic command @see me remote_open.
 |  On Windows, the installation of pycrypto can be tricky.
 |  See Pycrypto on Windows.
 |  Those modules are part of the Anaconda distribution.
 |
 |  Methods defined here:
 |
 |  __init__(self, server, username, password)
 |      constructor
 |
 |      @param      server      server
 |      @param      username    username
 |      @param      password    password
 |
 |  __str__(self)
 |      usual
 |
 |  close(self)
 |      close the connection
 |
 |  close_session(self)
 |      close a session
 |
 |  connect(self)
 |      connect
 |
 |  dfs_exists(self, path)
 |      tells if a file exists on the cluster
 |
 |      @param      path        path
 |      @return                 boolean
 |
 |      .. versionadded:: 1.1
 |
 |  dfs_ls(self, path)
 |      return the content of a folder on the cluster as a DataFrame
 |
 |      @param  path        path on the cluster
 |      @return             DataFrame
 |
 |      .. versionadded:: 1.1
 |
 |  dfs_mkdir(self, path)
 |      creates a directory on the cluster
 |
 |      @param      path        path
 |
 |      .. versionadded:: 1.1
 |
 |  dfs_rm(self, path, recursive=False)
 |      removes a file on the cluster
 |
 |      @param      path        path
 |      @param      recursive   boolean
 |
 |      .. versionadded:: 1.1
 |
 |  download(self, remotepath, localpath)
 |      download a file from the remote machine (not on the cluster)
 |      @param      localpath     local file
 |      @param      remotepath    remote file (it can be a list, localpath is a folder in that case)
 |
 |      .. versionchanged:: 1.1
 |          remotepath can be a list of paths
 |
 |  download_cluster(self, remotepath, localpath, merge=False)
 |      download a file directly from the cluster to the local machine
 |      @param      localpath       local file
 |      @param      remotepath      remote file (it can be a list, localpath is a folder in that case)
 |      @param      merge           True to use getmerge instead of get
 |
 |      .. versionadded:: 1.1
 |
 |  execute_command(self, command, no_exception=False, fill_stdin=None)
 |      execute a command line, it raises an error
 |      if there is an error
 |
 |      @param      command         command
 |      @param      no_exception    if True, do not raise any exception
 |      @param      fill_stdin      data to send on the stdin input
 |      @return                     stdout, stderr
 |
 |      Example of commands::
 |
 |          ssh.execute_command("ls")
 |          ssh.execute_command("hdfs dfs -ls")
 |
 |  exists(self, path)
 |      tells if a file exists on the bridge
 |
 |      @param      path        path
 |      @return                 boolean
 |
 |      .. versionadded:: 1.1
 |
 |  ls(self, path)
 |      return the content of a folder on the bridge as a DataFrame
 |
 |      @param  path        path on the bridge
 |      @return             DataFrame
 |
 |      .. versionadded:: 1.1
 |
 |  open_session(self, no_exception=False, timeout=1.0, add_eol=True, prompts=('~$', '>>>'), out_format=None)
 |      open a session with method invoke_shell
 |
 |      @param      no_exception    if True, do not raise any exception in case of error
 |      @param      timeout         timeout in s
 |      @param      add_eol         if True, the function will add a EOL to the sent command if it does not have one
 |      @param      prompts         if function terminates if the output ends by one of those strings.
 |      @param      out_format      None, plain, html
 |
 |      @example(How to open a remote shell?)
 |      @code
 |      ssh = ASSHClient(   "<server>",
 |                          "<login>",
 |                          "<password>")
 |      ssh.connect()
 |      out = ssh.send_recv_session("ls")
 |      print( ssh.send_recv_session("python") )
 |      print( ssh.send_recv_session("print('3')") )
 |      print( ssh.send_recv_session("import sysnsys.executable") )
 |      print( ssh.send_recv_session("sys.exit()") )
 |      print( ssh.send_recv_session(None) )
 |      ssh.close_session()
 |      ssh.close()
 |      @endcode
 |
 |      The notebook Communication with a remote Linux machine through SSH illustrates
 |      the output of these instructions.
 |
 |      @endexample
 |
 |  pig_submit(self, pig_file, dependencies=None, params=None, redirection='redirection', local=False, stop_on_failure=False, check=False, no_exception=True, fLOG=<function noLOG at 0x0000000008160E18>)
 |      submits a PIG script, it first upload the script
 |      to the default folder and submit it
 |
 |      @param      pig_file        pig script (local)
 |      @param      dependencies    others files to upload (still in the default folder)
 |      @param      params          parameters to send to the job
 |      @param      redirection     string empty or not
 |      @param      local           local run or not (option -x local)  (in that case, redirection will be empty)
 |      @param      stop_on_failure if True, add option -stop_on_failure on the command line
 |      @param      check           if True, add option -check (in that case, redirection will be empty)
 |      @param      no_exception    sent to @see me execute_command
 |      @param      fLOG            logging function
 |      @return                     out, err from @see me execute_command
 |
 |      If redirection is not empty, the job is submitted but
 |      the function returns after the standard output and error were
 |      redirected to redirection.out and redirection.err.
 |
 |      The first file will contain the results of commands
 |      DESCRIBE
 |      DUMP,
 |      EXPLAIN.
 |      The standard error receives logs and exceptions.
 |
 |      The function executes the command line::
 |
 |          pig  -execute -f <filename>
 |
 |      With redirection::
 |
 |          pig -execute -f <filename> 2> redirection.err 1> redirection.out &
 |
 |      .. versionadded:: 1.1
 |
 |  send_recv_session(self, fillin)
 |      Send something through a session,
 |      the function is supposed to return when the execute of the given command is done,
 |      but this is quite difficult to detect without knowing what exactly was send.
 |
 |      So we add a timeout just to tell the function it has to return even if nothing
 |      tells the command has finished. It fillin is None, the function will just
 |      listen to the output.
 |
 |      @param      fillin          sent to stdin
 |      @return                     stdout
 |
 |      The output contains
 |      escape codes.
 |      They can be converted to plain text or HTML
 |      by using the module ansiconv
 |      and ansi2html.
 |      This can be specified when opening the session.
 |
 |  upload(self, localpath, remotepath)
 |      upload a file to the remote machine (not on the cluster)
 |
 |      @param      localpath     local file (or a list of files)
 |      @param      remotepath    remote file
 |
 |      .. versionchanged:: 1.1
 |          it can upload multiple files if localpath is a list
 |
 |  upload_cluster(self, localpath, remotepath)
 |      the function directly uploads the file to the cluster, it first goes
 |      to the bridge, uploads it to the cluster and deletes it from the bridge
 |
 |      @param  localpath       local filename (or list of files)
 |      @param  remotepath      path to the cluster
 |      @return                 filename
 |
 |      .. versionadded:: 1.1
 |
 |  ----------------------------------------------------------------------
 |  Static methods defined here:
 |
 |  build_command_line_parameters(params)
 |      builds a string for pig based on the parameters in params
 |
 |      @param      params      dictionary
 |      @return                 string
 |
 |      .. versionadded:: 1.1
 |
 |  parse_lsout(out, local_schema=True)
 |      parses the output of a command ls
 |
 |      @param  out             output
 |      @param  local_schema    schema for the bridge or the cluster (False)
 |      @return                 DataFrame
 |
 |      .. versionadded:: 1.1
 |
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |
 |  __dict__
 |      dictionary for instance variables (if defined)
 |
 |  __weakref__
 |      list of weak references to the object (if defined)

We open the connection using the variables stored two cells above:

%remote_open
<pyenbc.remote.ssh_remote_connection.ASSHClient at 0xa854fd0>

We try a simple command:

%remote_cmd ls .
centrer_reduire.pig
ConfLongDemo_JSI.txt
diff_cluster
dummy
filter_example.pig
filter_example.redirect.err
filter_example.redirect.out
init_random.pig
iteration_complete.pig
nb_obervations.pig
pig_1436911046432.log
pig_1436913856496.log
pig_1436997076356.log
post_traitement.pig
pystream.pig
pystream.py
redirection.err
redirection.out
Skin_NonSkin.txt

We then execute a python program on the remote machine, we first same the following code as a program:

%%PYTHON exemple.py

import sys
print("path to python", sys.executable)
print("version ", sys.version_info)

The next command stores it as a file, uploads it to the remote machine and then executes it:

%remote_py exemple.py
('path to python', '/usr/bin/python')
('version ', (2, 6, 6, 'final', 0))

We check it is different from the local version:

import sys
sys.executable, sys.version_info
('c:\python34_x64\python.exe',
 sys.version_info(major=3, minor=4, micro=3, releaselevel='final', serial=0))

If you want to use a different version of the interpreter, you can try (not available here):

%remote_py -i=anaconda3/bin/python3.4 exemple.py
ERR:
bash: anaconda3/bin/python3.4: No such file or directory

OUT:

Sometimes, you need an interactive shell such as Putty. Let’s see how it works:

%open_remote_shell
True
%%shell_remote
ls
Last login: Fri Jan 30 11:37:57 2015 from aputeaux-551-1-113-111.w92-132.abo.wanadoo.fr
[xavierdupre@ws09-en1.tl: ~]$ ls
centrer_reduire.pig   diff_cluster  exemple.py          filter_example.redirect.err  init_random.pig         nb_obervations.pig     pig_1436913856496.log  post_traitement.pig  pystream.py      redirection.out
ConfLongDemo_JSI.txt  dummy         filter_example.pig  filter_example.redirect.out  iteration_complete.pig  pig_1436911046432.log  pig_1436997076356.log  pystream.pig         redirection.err  Skin_NonSkin.txt
[xavierdupre@ws09-en1.tl: ~]$
%shell_remote python
python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
%%shell_remote

import sys
sys.executable

>>> import sys
>>> sys.executable
'/usr/bin/python'
>>>

To close the shell, we can just type:

%close_remote_shell
True

And to close the connection:

%remote_close
True

END