2018-08-19 Install Python 3.7 and many packages on Linux Debian 9

Below you can find the instruction to get many modules on Debian 9 on Python 3.7. The first packages are quite common but the last instructions are needed to build all the content for my teachings. It begins with:

apt-get sudo
apt-get update

Install everything to generate documentation with Sphinx:

wget https://github.com/jgm/pandoc/releases/download/2.2.3.2/pandoc-2.2.3.2-1-amd64.deb
dpkg -i pandoc-2.2.3.2-1-amd64.deb
apt-get install texlive texlive-latex-extra texlive-xetex texlive-lang-european texlive-lang-french
apt-get install dvipng
apt-get install p7zip-full
apt-get install inkscape
apt-get install graphviz

Install git:

apt-get install git

Install Python dependencies:

apt-get install make build-essential libssl-dev zlib1g-dev
apt-get install libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm
apt-get install libncurses5-dev libncursesw5-dev xz-utils tk-dev

Install packages for keyring:

apt-get install libsecret-1-dev

Install and make Python, it must be installed in /usr/local otherwise virtualenv fails to create a vitual environment. See Building Python:

cd /usr/local
wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz
tar xzf Python-3.7.0.tgz
cd Python-3.7.0
./configure --enable-optimizations --with-ensurepip=install
make
make altinstall
chmod +x python3.7
cd ..
cd ~

Install pip:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
Python-3.7.0/python get-pip.py

Install packages:

curl https://raw.githubusercontent.com/sdpython/pyquickhelper/master/requirements.txt -o requirements.txt
./python -c "from pip._internal import main;main('install -r requirements.txt'.split())"

Uninstall packages:

./python -c "from pip._internal import main;main('uninstall jyquickhelper'.split())"

Intall java:

apt-get install default-jre default-jdk

Install Jenkins:

apt-get install daemon
wget https://pkg.jenkins.io/debian/binary/jenkins_2.138_all.deb
dpkg -i jenkins_2.138_all.deb

Install ufw:

apt-get install ufw

Allows connection to port 8080 (Jenkins default port):

ufw enable
ufw allow ssh
ufw allow 8080/tcp

Go to http://<ip_address>:8080 and follow the instructions, install also extensions Build-timeout Plugin, Console Column Plugin, Next Executions, Collapsing Console Sections Plugin, Credentials Plugin. Fix dependencies:

apt --fix-broken install

To reboot:

reboot

To complete Jenkins install, go to Default Jenkins User Password. You can use nano to edit. Install weird packages:

./python -c "from pip._internal import main;main('install git+https://github.com/fancycode/pylzma.git --upgrade --no-deps'.split())"
./python -c "from pip._internal import main;main('install git+https://github.com/sdpython/sphinxcontrib-images.git --upgrade --no-deps'.split())"

Setup one job on Jenkins with a Python script:

user = "<user>"
password = "<password>"

from pyquickhelper.jenkinshelper import JenkinsExt, setup_jenkins_server_yml, default_jenkins_jobs, default_engines
modules = default_jenkins_jobs()
engines = default_engines()
# or
# engines = dict(Python37="/home/ensaegithubxd/Python-3.7.0")
js = JenkinsExt('http://localhost:8080/', user, password, engines=engines)
setup_jenkins_server_yml(js, github="sdpython", modules=modules, fLOG=print, overwrite=True,
                         delete_first=False, location="somewhere/workspace")

Or if it is about pyquickhelper:

python setup.py jenkins <user> <password>

Checks that Jenkins user is present:

cut -d: -f1 /etc/passwd

Stores password with keyring. First switch to Jenkins user:

su
su - jenkins

Then:

import keyring
import os
keyring.set_password("web", "pyquickhelper,user", "...")
keyring.set_password("web", "pyquickhelper,pwd", "...")
mkdir -p /var/lib/jenkins/workspace/local_pypi/local_pypi_server

Additional dependencies unrar, nodejs:

apt-get install unrar
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
apt-get install -y nodejs

Additional dependencies geos, geopandas (current version of pyproj does not compile for Python 3.7, the latest sources must be used):

apt-get install libproj-dev proj-bin proj-data
pip3.7 install cython
pip3.7 install git+https://github.com/jswhit/pyproj.git
apt-get install libc6
apt-get install libgeos-dev
apt-get install python-gdal gdal-bin
export PYPROJ_VERSION=4.9.3-2
wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj9_$PYPROJ_VERSION_amd64.deb
dpkg -i libproj9_$PYPROJ_VERSION_amd64.deb
wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj-dev_$PYPROJ_VERSION_amd64.deb
dpkg -i libproj-dev_$PYPROJ_VERSION_amd64.deb
pip3.7 install git+https://github.com/jswhit/pyproj.git
pip3.7 install geopandas

Install avconv for ffmpeg :

apt-get install libav-tools

Install protobuf:

apt-get -y install protobuf-compiler

Other missing dependencies:

pip3.7 install nltk statsmodels cartopy imbalanced-learn seaborn gensim descartes networkx
pip3.7 install bqplot memory_profiler xarray folium antlr4-python3-runtime
pip3.7 install category_encoders --no-deps
pip3.7 install torchvision chainer fcn
pip3.7 install spacy
pip3.7 install xlwt xlrd wikipedia pycurl xlsxwriter pyinstrument wptools flask dask bcolz

For tables :

apt-get install libhdf5-dev libblosc-dev liblzo2-dev
pip3.7 install tables

Optional dependencies:

pip3.7 install sphinxcontrib.blockdiag sphinx_bootstrap_theme moviepy mutagen vprof solar-theme guzzle_sphinx_theme
pip3.7 install sphinxjp.themes.basicstrap sphinx_readable_theme
pip3.7 install pyspark qgrid yahoo_historical py4j bayespy lightgbm
pip3.7 install version_information xgboost natsort pivottablejs prettytable
pip3.7 install colormap easydev dbfread quandl seaborn requests-file requests-ftp lxml opencv-python python-pptx
pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fairtest-0.1-py3-none-any.whl
pip3.7 install reportlab sympy vispy cufflinks brythonmagic pydy seasonal
pip3.7 install treeinterpreter tutormagic pympler selenium httplib2
pip3.7 install tpot pygal google-api-python-client datashader PyQt5
pip3.7 install pyecharts pygal-maps-fr pythreejs vega yahoo-historical lifelines fast-histogram geoplotlib
pip3.7 install missingno mpl-scatter-density mizani lml macropy3 plotnine geopy zarr
pip3.7 install jsonpickle forestci psycopg2

apt-get install  libappindicator3-1
apt --fix-broken install
apt-get install  libappindicator3-1
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
apt-get install chromium-driver

For spacy:

pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fr_core_news_sm-2.0.0-py3-none-any.whl

For module imageio, the second instruction must be executed with every disinct user once:

pip3.7 install imageio
python3.7 -c "import imageio;imageio.plugins.ffmpeg.download()"

For pyaudio:

apt-get install portaudio19-dev

pip3.7 install pyaudio

For pycurl :

apt-get install libcurl4-openssl-dev
pip3.7 install pycurl

Install mono (for pythonnet):

apt-get install apt-transport-https
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
apt-get update
apt-get install mono-devel ca-certificates-mono

And for dotnet (see Install .NET Core SDK on Linux Debian 9):

wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/debian/9/prod.list
mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
chown root:root /etc/apt/sources.list.d/microsoft-prod.list

apt-get update
apt-get install dotnet-sdk-2.1

Deal with permissions issues for nugets:

chown -R jenkins:jenkins /tmp/NuGetScratch/

Install clang 6 (for pythonnet):

echo "deb http://ftp.debian.org/debian stretch-backports main" | sudo tee /etc/apt/sources.list.d/clang-stable.list
apt-get update
apt-get install clang-6.0
alias clang='clang-6.0'

Install line_profiler :

pip3.7 install git+https://github.com/rkern/line_profiler.git

Install pythonnet:

git clone -b py37 https://github.com/pythonnet/pythonnet.git
cd pythonnet
python3.7 setup.py install --xplat

For ML.net (see Building ML.NET on Linux and macOS):

apt-get update
apt-get install libunwind8

You might need to run chmod 777 -R /tmp/NuGetScratch;chmod 777 -R /tmp/NuGet. I got the following issue Restore task fails in an multi user scenario <https://github.com/NuGet/Home/issues/3897>_.

For cvxopt :

apt-get install libblas-dev liblapack-dev libsuitesparse-dev
pip3.7 install git+https://github.com/cvxopt/cvxopt.git@dev

For pygame:

apt-get install libasound2-dev
apt-get install libx11-dev
apt-get -y install xvfb pulseaudio
apt-get install -y python-pygame
pip3.7 install pygame

For minepy :

pip3.7 install git+https://github.com/sdpython/minepy.git

For onnx:

apt-get install cmake
pip3.7 install onnx

MKL and deep learning

The machine I used does not have GPU. Missing MKL : MKL for .deb-based systems: An easy recipe.

About pyopencl …

# apt-get install ocl-icd-opencl-dev
apt-get install beignet-opencl-icd
apt-get install libpocl-dev
pip3.7 install pyopencl

About pycuda :

apt-get install nvidia-cuda-toolkit
pip3.7 install pycuda

Misc:

pip3.7 install git+https://github.com/sdpython/pyPdf --upgrade --no-deps

About TensorFlow, it does not compile yet on Python 3.7, see pull request Quick Fix for Python 3.7.

git clone https://github.com/tensorflow/tensorflow

For Azure :

pip3.7 install azure

Install a newer version of cmake:

wget https://cmake.org/files/v3.12/cmake-3.12.1.tar.gz
tar xzf cmake-3.12.1
./configure
make