.. _exampleofsshclientcommunicationrst: ===================================================== Communication with a remote Linux machine through SSH ===================================================== .. only:: html **Links:** :download:`notebook `, :downloadlink:`html `, :download:`PDF `, :download:`python `, :downloadlink:`slides `, :githublink:`GitHub|_doc/notebooks/example_of_ssh_client_communication.ipynb|*` In this notebook, we show how to communicate with a remote machine. This machine was set up using Ubuntu 12 (and with `Azure `__ : `Create a Virtual Machine Running Linux `__). The following code is used to get the credentials without printing them on the notebook. .. code:: ipython3 %load_ext pyensae %load_ext pyenbc .. code:: ipython3 import pyquickhelper.ipythonhelper as ipy params={"server":"", "username":"", "password":""} ipy.open_html_form(params=params,title="credentials",key_save="ssh_remote") .. raw:: html
credentials
password
server
username
.. code:: ipython3 password = ssh_remote["password"] server = ssh_remote["server"] username = ssh_remote["username"] We are going to use the following magic commands `MagicRemote `__: .. code:: ipython3 from pyense.remote import ASSHClient help(ASSHClient) .. parsed-literal:: Help on class ASSHClient in module pyenbc.remote.ssh_remote_connection: class ASSHClient(builtins.object) | A simple class to access to remote machine through SSH. | It requires modules | `paramiko `_, | `pycrypto `_, | `ecdsa `_. | | This class is used in magic command @see me remote_open. | On Windows, the installation of pycrypto can be tricky. | See `Pycrypto on Windows `_. | Those modules are part of the `Anaconda `_ distribution. | | Methods defined here: | | __init__(self, server, username, password) | constructor | | @param server server | @param username username | @param password password | | __str__(self) | usual | | close(self) | close the connection | | close_session(self) | close a session | | connect(self) | connect | | dfs_exists(self, path) | tells if a file exists on the cluster | | @param path path | @return boolean | | .. versionadded:: 1.1 | | dfs_ls(self, path) | return the content of a folder on the cluster as a DataFrame | | @param path path on the cluster | @return DataFrame | | .. versionadded:: 1.1 | | dfs_mkdir(self, path) | creates a directory on the cluster | | @param path path | | .. versionadded:: 1.1 | | dfs_rm(self, path, recursive=False) | removes a file on the cluster | | @param path path | @param recursive boolean | | .. versionadded:: 1.1 | | download(self, remotepath, localpath) | download a file from the remote machine (not on the cluster) | @param localpath local file | @param remotepath remote file (it can be a list, localpath is a folder in that case) | | .. versionchanged:: 1.1 | remotepath can be a list of paths | | download_cluster(self, remotepath, localpath, merge=False) | download a file directly from the cluster to the local machine | @param localpath local file | @param remotepath remote file (it can be a list, localpath is a folder in that case) | @param merge True to use getmerge instead of get | | .. versionadded:: 1.1 | | execute_command(self, command, no_exception=False, fill_stdin=None) | execute a command line, it raises an error | if there is an error | | @param command command | @param no_exception if True, do not raise any exception | @param fill_stdin data to send on the stdin input | @return stdout, stderr | | Example of commands:: | | ssh.execute_command("ls") | ssh.execute_command("hdfs dfs -ls") | | exists(self, path) | tells if a file exists on the bridge | | @param path path | @return boolean | | .. versionadded:: 1.1 | | ls(self, path) | return the content of a folder on the bridge as a DataFrame | | @param path path on the bridge | @return DataFrame | | .. versionadded:: 1.1 | | open_session(self, no_exception=False, timeout=1.0, add_eol=True, prompts=('~$', '>>>'), out_format=None) | open a session with method `invoke_shell `_ | | @param no_exception if True, do not raise any exception in case of error | @param timeout timeout in s | @param add_eol if True, the function will add a EOL to the sent command if it does not have one | @param prompts if function terminates if the output ends by one of those strings. | @param out_format None, plain, html | | @example(How to open a remote shell?) | @code | ssh = ASSHClient( "", | "", | "") | ssh.connect() | out = ssh.send_recv_session("ls") | print( ssh.send_recv_session("python") ) | print( ssh.send_recv_session("print('3')") ) | print( ssh.send_recv_session("import sys\nsys.executable") ) | print( ssh.send_recv_session("sys.exit()") ) | print( ssh.send_recv_session(None) ) | ssh.close_session() | ssh.close() | @endcode | | The notebook :ref:`exampleofsshclientcommunicationrst` illustrates | the output of these instructions. | | @endexample | | pig_submit(self, pig_file, dependencies=None, params=None, redirection='redirection', local=False, stop_on_failure=False, check=False, no_exception=True, fLOG=) | submits a PIG script, it first upload the script | to the default folder and submit it | | @param pig_file pig script (local) | @param dependencies others files to upload (still in the default folder) | @param params parameters to send to the job | @param redirection string empty or not | @param local local run or not (option `-x local `_) (in that case, redirection will be empty) | @param stop_on_failure if True, add option ``-stop_on_failure`` on the command line | @param check if True, add option ``-check`` (in that case, redirection will be empty) | @param no_exception sent to @see me execute_command | @param fLOG logging function | @return out, err from @see me execute_command | | If *redirection* is not empty, the job is submitted but | the function returns after the standard output and error were | redirected to ``redirection.out`` and ``redirection.err``. | | The first file will contain the results of commands | `DESCRIBE `_ | `DUMP `_, | `EXPLAIN `_. | The standard error receives logs and exceptions. | | The function executes the command line:: | | pig -execute -f | | With redirection:: | | pig -execute -f 2> redirection.err 1> redirection.out & | | .. versionadded:: 1.1 | | send_recv_session(self, fillin) | Send something through a session, | the function is supposed to return when the execute of the given command is done, | but this is quite difficult to detect without knowing what exactly was send. | | So we add a timeout just to tell the function it has to return even if nothing | tells the command has finished. It fillin is None, the function will just | listen to the output. | | @param fillin sent to stdin | @return stdout | | The output contains | `escape codes `_. | They can be converted to plain text or HTML | by using the module `ansiconv `_ | and `ansi2html `_. | This can be specified when opening the session. | | upload(self, localpath, remotepath) | upload a file to the remote machine (not on the cluster) | | @param localpath local file (or a list of files) | @param remotepath remote file | | .. versionchanged:: 1.1 | it can upload multiple files if localpath is a list | | upload_cluster(self, localpath, remotepath) | the function directly uploads the file to the cluster, it first goes | to the bridge, uploads it to the cluster and deletes it from the bridge | | @param localpath local filename (or list of files) | @param remotepath path to the cluster | @return filename | | .. versionadded:: 1.1 | | ---------------------------------------------------------------------- | Static methods defined here: | | build_command_line_parameters(params) | builds a string for ``pig`` based on the parameters in params | | @param params dictionary | @return string | | .. versionadded:: 1.1 | | parse_lsout(out, local_schema=True) | parses the output of a command ls | | @param out output | @param local_schema schema for the bridge or the cluster (False) | @return DataFrame | | .. versionadded:: 1.1 | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) We open the connection using the variables stored two cells above: .. code:: ipython3 %remote_open .. parsed-literal:: We try a simple command: .. code:: ipython3 %remote_cmd ls . .. raw:: html
    centrer_reduire.pig
    ConfLongDemo_JSI.txt
    diff_cluster
    dummy
    filter_example.pig
    filter_example.redirect.err
    filter_example.redirect.out
    init_random.pig
    iteration_complete.pig
    nb_obervations.pig
    pig_1436911046432.log
    pig_1436913856496.log
    pig_1436997076356.log
    post_traitement.pig
    pystream.pig
    pystream.py
    redirection.err
    redirection.out
    Skin_NonSkin.txt

    
We then execute a python program on the remote machine, we first same the following code as a program: .. code:: ipython3 %%PYTHON exemple.py import sys print("path to python", sys.executable) print("version ", sys.version_info) The next command stores it as a file, uploads it to the remote machine and then executes it: .. code:: ipython3 %remote_py exemple.py .. raw:: html
    ('path to python', '/usr/bin/python')
    ('version ', (2, 6, 6, 'final', 0))

    
We check it is different from the local version: .. code:: ipython3 import sys sys.executable, sys.version_info .. parsed-literal:: ('c:\\python34_x64\\python.exe', sys.version_info(major=3, minor=4, micro=3, releaselevel='final', serial=0)) If you want to use a different version of the interpreter, you can try (not available here): .. code:: ipython3 %remote_py -i=anaconda3/bin/python3.4 exemple.py .. raw:: html ERR:
    bash: anaconda3/bin/python3.4: No such file or directory

    
OUT:

    
Sometimes, you need an interactive shell such as `Putty `__. Let’s see how it works: .. code:: ipython3 %open_remote_shell .. parsed-literal:: True .. code:: ipython3 %%shell_remote ls .. raw:: html
    Last login: Fri Jan 30 11:37:57 2015 from aputeaux-551-1-113-111.w92-132.abo.wanadoo.fr
    [xavierdupre@ws09-en1.tl: ~]$ ls
    centrer_reduire.pig   diff_cluster  exemple.py          filter_example.redirect.err  init_random.pig         nb_obervations.pig     pig_1436913856496.log  post_traitement.pig  pystream.py      redirection.out
    ConfLongDemo_JSI.txt  dummy         filter_example.pig  filter_example.redirect.out  iteration_complete.pig  pig_1436911046432.log  pig_1436997076356.log  pystream.pig         redirection.err  Skin_NonSkin.txt
    [xavierdupre@ws09-en1.tl: ~]$ 
    
.. code:: ipython3 %shell_remote python .. raw:: html
    python
    Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36) 
    [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> 
    
.. code:: ipython3 %%shell_remote import sys sys.executable .. raw:: html

    >>> import sys
    >>> sys.executable
    '/usr/bin/python'
    >>> 
    
To close the shell, we can just type: .. code:: ipython3 %close_remote_shell .. parsed-literal:: True And to close the connection: .. code:: ipython3 %remote_close .. parsed-literal:: True **END**