.. blogpost::
    :title: Install Python 3.7 and many packages on Linux Debian 9
    :keywords: python 3.7
    :date: 2018-08-19
    :categories: python
    :lid: blog-2018-python37-linux

    Below you can find the instruction to get many modules
    on :epkg:`Debian 9` on :epkg:`Python` 3.7.
    The first packages are quite common but the last instructions
    are needed to build all the content for my teachings.
    It begins with:

    ::

        apt-get sudo
        apt-get update

    Install everything to generate documentation with :epkg:`Sphinx`:

    ::

        wget https://github.com/jgm/pandoc/releases/download/2.2.3.2/pandoc-2.2.3.2-1-amd64.deb
        dpkg -i pandoc-2.2.3.2-1-amd64.deb
        apt-get install texlive texlive-latex-extra texlive-xetex texlive-lang-european texlive-lang-french
        apt-get install dvipng
        apt-get install p7zip-full
        apt-get install inkscape
        apt-get install graphviz

    Install :epkg:`git`:

    ::

        apt-get install git

    Install :epkg:`Python` dependencies:

    ::

        apt-get install make build-essential libssl-dev zlib1g-dev
        apt-get install libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm
        apt-get install libncurses5-dev libncursesw5-dev xz-utils tk-dev

    Install packages for :epkg:`keyring`:

    ::

        apt-get install libsecret-1-dev

    Install and make :epkg:`Python`,
    it must be installed in ``/usr/local`` otherwise
    :epkg:`virtualenv` fails to create a vitual
    environment.
    See `Building Python <https://docs.python.org/3/using/unix.html#building-python>`_:

    ::

        cd /usr/local
        wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz
        tar xzf Python-3.7.0.tgz
        cd Python-3.7.0
        ./configure --enable-optimizations --with-ensurepip=install
        make
        make altinstall
        chmod +x python3.7
        cd ..
        cd ~

    Install :epkg:`pip`:

    ::

        curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
        Python-3.7.0/python get-pip.py

    Install packages:

    ::

        curl https://raw.githubusercontent.com/sdpython/pyquickhelper/master/requirements.txt -o requirements.txt
        ./python -c "from pip._internal import main;main('install -r requirements.txt'.split())"

    Uninstall packages:

    ::

        ./python -c "from pip._internal import main;main('uninstall jyquickhelper'.split())"

    Intall :epkg:`java`:

    ::

        apt-get install default-jre default-jdk

    Install :epkg:`Jenkins`:

    ::

        apt-get install daemon
        wget https://pkg.jenkins.io/debian/binary/jenkins_2.138_all.deb
        dpkg -i jenkins_2.138_all.deb

    Install :epkg:`ufw`:

    ::

        apt-get install ufw

    Allows connection to port 8080 (:epkg:`Jenkins` default port):

    ::

        ufw enable
        ufw allow ssh
        ufw allow 8080/tcp

    Go to ``http://<ip_address>:8080`` and follow the instructions,
    install also extensions
    `Build-timeout Plugin <https://wiki.jenkins.io/display/JENKINS/Build-timeout+Plugin>`_,
    `Console Column Plugin <https://wiki.jenkins.io/display/JENKINS/Console+Column+Plugin>`_,
    `Next Executions <https://wiki.jenkins.io/display/JENKINS/Next+Executions>`_,
    `Collapsing Console Sections Plugin <https://wiki.jenkins.io/display/JENKINS/Collapsing+Console+Sections+Plugin>`_,
    `Credentials Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Plugin>`_.
    Fix dependencies:

    ::

        apt --fix-broken install

    To reboot:

    ::

        reboot

    To complete :epkg:`Jenkins` install, go to
    `Default Jenkins User Password <https://stackoverflow.com/questions/25041125/default-jenkins-user-password>`_.
    You can use `nano <https://www.nano-editor.org/>`_
    to edit. Install weird packages:

    ::

        ./python -c "from pip._internal import main;main('install git+https://github.com/fancycode/pylzma.git --upgrade --no-deps'.split())"
        ./python -c "from pip._internal import main;main('install git+https://github.com/sdpython/sphinxcontrib-images.git --upgrade --no-deps'.split())"

    Setup one job on :epkg:`Jenkins`
    with a :epkg:`Python` script:

    ::

        user = "<user>"
        password = "<password>"

        from pyquickhelper.jenkinshelper import JenkinsExt, setup_jenkins_server_yml, default_jenkins_jobs, default_engines
        modules = default_jenkins_jobs()
        engines = default_engines()
        # or
        # engines = dict(Python37="/home/ensaegithubxd/Python-3.7.0")
        js = JenkinsExt('http://localhost:8080/', user, password, engines=engines)
        setup_jenkins_server_yml(js, github="sdpython", modules=modules, fLOG=print, overwrite=True,
                                 delete_first=False, location="somewhere/workspace")

    Or if it is about :epkg:`pyquickhelper`:

    ::

        python setup.py jenkins <user> <password>

    Checks that :epkg:`Jenkins` user is present:

    ::

        cut -d: -f1 /etc/passwd

    Stores password with :epkg:`keyring`.
    First switch to :epkg:`Jenkins` user:

    ::

        su
        su - jenkins

    Then:

    ::

        import keyring
        import os
        keyring.set_password("web", "pyquickhelper,user", "...")
        keyring.set_password("web", "pyquickhelper,pwd", "...")
        mkdir -p somewhere/workspace/local_pypi/local_pypi_server

    Additional dependencies :epkg:`unrar`, :epkg:`nodejs`:

    ::

        apt-get install unrar
        curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
        apt-get install -y nodejs

    Additional dependencies *geos*, *geopandas*
    (current version of *pyproj* does not compile
    for Python 3.7, the latest sources must be used):

    ::

        apt-get install libproj-dev proj-bin proj-data
        pip3.7 install cython
        pip3.7 install git+https://github.com/jswhit/pyproj.git
        apt-get install libc6
        apt-get install libgeos-dev
        apt-get install python-gdal gdal-bin
        export PYPROJ_VERSION=4.9.3-2
        wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj9_$PYPROJ_VERSION_amd64.deb
        dpkg -i libproj9_$PYPROJ_VERSION_amd64.deb
        wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj-dev_$PYPROJ_VERSION_amd64.deb
        dpkg -i libproj-dev_$PYPROJ_VERSION_amd64.deb
        pip3.7 install git+https://github.com/jswhit/pyproj.git
        pip3.7 install geopandas

    Install :epkg:`avconv` for :epkg:`ffmpeg` :

    ::

        apt-get install libav-tools

    Install :epkg:`protobuf`:

    ::

        apt-get -y install protobuf-compiler

    Other missing dependencies:

    ::

        pip3.7 install nltk statsmodels cartopy imbalanced-learn seaborn gensim descartes networkx
        pip3.7 install bqplot memory_profiler xarray folium antlr4-python3-runtime
        pip3.7 install category_encoders --no-deps
        pip3.7 install torchvision chainer fcn
        pip3.7 install spacy
        pip3.7 install xlwt xlrd wikipedia pycurl xlsxwriter pyinstrument wptools flask dask bcolz

    For :epkg:`tables` :

    ::

        apt-get install libhdf5-dev libblosc-dev liblzo2-dev
        pip3.7 install tables

    Optional dependencies:

    ::

        pip3.7 install sphinxcontrib.blockdiag sphinx_bootstrap_theme moviepy mutagen vprof solar-theme guzzle_sphinx_theme
        pip3.7 install sphinxjp.themes.basicstrap sphinx_readable_theme
        pip3.7 install pyspark qgrid yahoo_historical py4j bayespy lightgbm
        pip3.7 install version_information xgboost natsort pivottablejs prettytable
        pip3.7 install colormap easydev dbfread quandl seaborn requests-file requests-ftp lxml opencv-python python-pptx
        pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fairtest-0.1-py3-none-any.whl
        pip3.7 install reportlab sympy vispy cufflinks brythonmagic pydy seasonal
        pip3.7 install treeinterpreter tutormagic pympler selenium httplib2
        pip3.7 install tpot pygal google-api-python-client datashader PyQt5
        pip3.7 install pyecharts pygal-maps-fr pythreejs vega yahoo-historical lifelines fast-histogram geoplotlib
        pip3.7 install missingno mpl-scatter-density mizani lml macropy3 plotnine geopy zarr
        pip3.7 install jsonpickle forestci psycopg2

        apt-get install  libappindicator3-1
        apt --fix-broken install
        apt-get install  libappindicator3-1
        wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
        dpkg -i google-chrome-stable_current_amd64.deb
        apt-get install chromium-driver

    For :epkg:`spacy`:

    ::

        pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fr_core_news_sm-2.0.0-py3-none-any.whl

    For module :epkg:`imageio`, the second instruction must be executed
    with every disinct user once:

    ::

        pip3.7 install imageio
        python3.7 -c "import imageio;imageio.plugins.ffmpeg.download()"

    For :epkg:`pyaudio`:

    ::

        apt-get install portaudio19-dev

        pip3.7 install pyaudio

    For :epkg:`pycurl` :

    ::

        apt-get install libcurl4-openssl-dev
        pip3.7 install pycurl

    Install :epkg:`mono` (for :epkg:`pythonnet`):

    ::

        apt-get install apt-transport-https
        apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
        echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
        apt-get update
        apt-get install mono-devel ca-certificates-mono

    And for :epkg:`dotnet`
    (see `Install .NET Core SDK on Linux Debian 9 <https://www.microsoft.com/net/download/linux-package-manager/debian9/sdk-current>`_):

    ::

        wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
        mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
        wget -q https://packages.microsoft.com/config/debian/9/prod.list
        mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
        chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
        chown root:root /etc/apt/sources.list.d/microsoft-prod.list

        apt-get update
        apt-get install dotnet-sdk-2.1

    Deal with permissions issues for nugets:

    ::

        chown -R jenkins:jenkins /tmp/NuGetScratch/

    Install :epkg:`clang` 6 (for :epkg:`pythonnet`):

    ::

        echo "deb http://ftp.debian.org/debian stretch-backports main" | sudo tee /etc/apt/sources.list.d/clang-stable.list
        apt-get update
        apt-get install clang-6.0
        alias clang='clang-6.0'

    Install :epkg:`line_profiler` :

    ::

        pip3.7 install git+https://github.com/rkern/line_profiler.git

    Install :epkg:`pythonnet`:

    ::

        git clone -b py37 https://github.com/pythonnet/pythonnet.git
        cd pythonnet
        python3.7 setup.py install --xplat

    For :epkg:`ML.net`
    (see `Building ML.NET on Linux and macOS <https://github.com/dotnet/machinelearning/blob/master/docs/building/unix-instructions.md>`_):

    ::

        apt-get update
        apt-get install libunwind8

    You might need to run ``chmod 777 -R /tmp/NuGetScratch;chmod 777 -R /tmp/NuGet``.
    I got the following issue
    `Restore task fails in an multi user scenario <https://github.com/NuGet/Home/issues/3897>_`.

    For :epkg:`cvxopt` :

    ::

        apt-get install libblas-dev liblapack-dev libsuitesparse-dev
        pip3.7 install git+https://github.com/cvxopt/cvxopt.git@dev

    For :epkg:`pygame`:

    ::

        apt-get install libasound2-dev
        apt-get install libx11-dev
        apt-get -y install xvfb pulseaudio
        apt-get install -y python-pygame
        pip3.7 install pygame

    For :epkg:`minepy` :

    ::

        pip3.7 install git+https://github.com/sdpython/minepy.git

    For :epkg:`onnx`:

    ::

        apt-get install cmake
        pip3.7 install onnx

    **MKL** and **deep learning**

    The machine I used does not have GPU.
    Missing :epkg:`MKL` :
    `MKL for .deb-based systems: An easy recipe <https://github.com/eddelbuettel/mkl4deb>`_.

    About :epkg:`pyopencl` ...

    ::

        # apt-get install ocl-icd-opencl-dev
        apt-get install beignet-opencl-icd
        apt-get install libpocl-dev
        pip3.7 install pyopencl

    About :epkg:`pycuda` :

    ::

        apt-get install nvidia-cuda-toolkit
        pip3.7 install pycuda

    Misc:

    ::

        pip3.7 install git+https://github.com/sdpython/pyPdf --upgrade --no-deps

    About :epkg:`TensorFlow`, it does not compile yet
    on :epkg:`Python` 3.7, see pull request
    `Quick Fix for Python 3.7 <https://github.com/tensorflow/tensorflow/pull/21202>`_.

    ::

        git clone https://github.com/tensorflow/tensorflow

    For :epkg:`Azure` :

    ::

        pip3.7 install azure

    Install a newer version of :epkg:`cmake`:

    ::

        wget https://cmake.org/files/v3.12/cmake-3.12.1.tar.gz
        tar xzf cmake-3.12.1
        ./configure
        make