.. blogpost:: :title: Install Python 3.7 and many packages on Linux Debian 9 :keywords: python 3.7, debian :date: 2018-12-29 :categories: python :lid: blog-2018-python37-linux2 Same steps four months later: :ref:`Install Python 3.7 and many packages on Linux Debian 9 `. :: apt-get install sudo -y apt-get update Install everything to generate documentation with :epkg:`Sphinx`: :: wget https://github.com/jgm/pandoc/releases/download/2.5/pandoc-2.5-1-amd64.deb dpkg -i pandoc-2.5-1-amd64.deb apt-get install -y texlive texlive-latex-extra texlive-xetex texlive-lang-european texlive-lang-french apt-get install -y dvipng p7zip-full inkscape graphviz git uuid-dev apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm apt-get install -y libncurses5-dev libncursesw5-dev xz-utils tk-dev libsecret-1-dev libffi-dev curl apt-get install -y liblzma-doc liblzma-dev libgdbm-dev libsqlite3-dev libdb5.3-dev nano apt-get install -y libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev apt-get install -y libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev libffi-dev apt-get install -y libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev apt-get install -y unrar libstdc++ *MKL*, see `MKL for .deb-based systems: An easy recipe `_. :: cd /tmp wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list' apt-get update apt-get install intel-mkl-64bit-2018.2-046 update-alternatives --install /usr/lib/x86_64-linux-gnu/libblas.so \ libblas.so-x86_64-linux-gnu /opt/intel/mkl/lib/intel64/libmkl_rt.so 50 update-alternatives --install /usr/lib/x86_64-linux-gnu/libblas.so.3 \ libblas.so.3-x86_64-linux-gnu /opt/intel/mkl/lib/intel64/libmkl_rt.so 50 update-alternatives --install /usr/lib/x86_64-linux-gnu/liblapack.so \ liblapack.so-x86_64-linux-gnu /opt/intel/mkl/lib/intel64/libmkl_rt.so 50 update-alternatives --install /usr/lib/x86_64-linux-gnu/liblapack.so.3 \ liblapack.so.3-x86_64-linux-gnu /opt/intel/mkl/lib/intel64/libmkl_rt.so 50 echo "/opt/intel/lib/intel64" > /etc/ld.so.conf.d/mkl.conf echo "/opt/intel/mkl/lib/intel64" >> /etc/ld.so.conf.d/mkl.conf ldconfig echo "MKL_THREADING_LAYER=GNU" >> /etc/environment Install a newer version of :epkg:`cmake`: :: wget https://cmake.org/files/v3.13/cmake-3.13.2.tar.gz tar xzf cmake-3.13.2.tar.gz cd cmake-3.13.2 ./configure make make install Install and make :epkg:`Python`, it must be installed in ``/usr/local`` otherwise :epkg:`virtualenv` fails to create a vitual environment. See `Building Python `_: :: cd /usr/local wget https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz tar xzf Python-3.7.2.tgz cd Python-3.7.2 ./configure --enable-optimizations --with-ensurepip=install make make altinstall chmod +x python3.7 curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py ./python get-pip.py cd .. cd ~ Some package require a different build because they need shared libraries: :: ./configure --enable-optimizations --with-ensurepip=install --enable-shared --prefix=/opt/bin About ``--enable-shared``, see `Relocation R_X86_64_32S against '_Py_NotImplementedStruct' can not be used when making a shared object; recompile with -fPIC `_, `How to configure python with --enable-shared on Ubuntu 18.04 `_. Option ``--enable-shared --prefix=/opt/python`` is needed when to embed *python* interpreter in a C application, see `Embedding the Python interpreter `_. Environment variable ``LD_LIBRARY_PATH`` must be set: :: export LD_LIBRARY_PATH=/opt/bin Add python to ``PATH``: :: export PATH=/usr/local/Python-3.7.2:$PATH Install packages: :: curl https://raw.githubusercontent.com/sdpython/pyquickhelper/master/requirements.txt -o requirements.txt python3.7 -c "from pip._internal import main;main('install -r requirements.txt'.split())" python3.7 -c "from pip._internal import main;main('uninstall -y jyquickhelper'.split())" Intall :epkg:`java`, :epkg:`Jenkins`, :epkg:`ufw`: :: apt-get install -y default-jre default-jdk daemon psmisc wget https://pkg.jenkins.io/debian/binary/jenkins_2.156_all.deb dpkg -i jenkins_2.156_all.deb apt-get install -y ufw Allows connection to port 8080 (:epkg:`Jenkins` default port): :: ufw enable ufw allow ssh ufw allow 8080/tcp Go to ``http://:8080`` and follow the instructions, install also extensions `Build-timeout Plugin `_, `Console Column Plugin `_, `Next Executions `_, `Collapsing Console Sections Plugin `_, `Credentials Plugin `_. If you need to manually restart :epkg:`Jenkins`, go to `How to restart Jenkins manually? `_. Fix dependencies: :: apt --fix-broken install Additional dependencies :epkg:`unrar`, :epkg:`nodejs`: :: curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - apt-get install -y nodejs Additional dependencies *geos*, *geopandas* (current version of *pyproj* does not compile for Python 3.7, the latest sources must be used): :: apt-get install -y libproj-dev proj-bin proj-data libc6 libgeos-dev python-gdal gdal-bin export PYPROJMAJVER=13 export PYPROJVERSION=5.2.0-1 wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj${PYPROJMAJVER}_${PYPROJVERSION}_amd64.deb dpkg -i libproj${PYPROJMAJVER}_${PYPROJVERSION}_amd64.deb wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj-dev_${PYPROJVERSION}_amd64.deb dpkg -i libproj-dev_${PYPROJVERSION}_amd64.deb apt-get install -y libav-tools protobuf-compiler libprotoc-dev libcurl4-openssl-dev libssl-dev apt-get install -y libhdf5-dev libblosc-dev liblzo2-dev libappindicator3-1 fonts-liberation apt --fix-broken install wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb dpkg -i google-chrome-stable_current_amd64.deb apt-get install -y chromium-driver apt-get install -y portaudio19-dev libblas-dev liblapack-dev libsuitesparse-dev apt-get install -y libasound2-dev libx11-dev xvfb pulseaudio python-pygame libunwind8 Other missing dependencies: :: pip3.7 install cython pip3.7 install git+https://github.com/jswhit/pyproj.git pip3.7 install geopandas descartes pip3.7 install tables nltk statsmodels cartopy imbalanced-learn seaborn gensim networkx pip3.7 install bqplot memory_profiler xarray folium antlr4-python3-runtime pip3.7 install category_encoders --no-deps pip3.7 install torchvision chainer fcn pip3.7 install spacy pip3.7 install xlwt xlrd wikipedia pycurl xlsxwriter pyinstrument wptools flask dask bcolz pip3.7 install sphinxcontrib.blockdiag sphinx_bootstrap_theme moviepy mutagen vprof solar-theme guzzle_sphinx_theme sphinx_modern_theme_modified sphinxjp.themes.basicstrap sphinx_readable_theme pip3.7 install pyspark qgrid yahoo_historical py4j bayespy lightgbm pip3.7 install version_information xgboost natsort pivottablejs prettytable pip3.7 install colormap easydev dbfread quandl seaborn requests-file requests-ftp lxml opencv-python python-pptx pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fairtest-0.1-py3-none-any.whl pip3.7 install reportlab sympy vispy cufflinks brythonmagic pydy seasonal pip3.7 install treeinterpreter tutormagic pympler selenium httplib2 pip3.7 install tpot pygal google-api-python-client datashader PyQt5 pip3.7 install pyecharts pygal-maps-fr pythreejs vega yahoo-historical lifelines fast-histogram geoplotlib pip3.7 install missingno mpl-scatter-density mizani lml macropy3 plotnine geopy zarr pip3.7 install jsonpickle forestci psycopg2 pypi-server jupyterlab pip3.7 install imageio pyaudio pygame pybind11 pip3.7 install git+https://github.com/cvxopt/cvxopt.git@dev pip3.7 install mpl-finance pip3.7 install git+https://github.com/sdpython/minepy.git pip3.7 install git+https://github.com/sdpython/pyPdf --upgrade --no-deps :: pip3.7 install matplotlib pandas numpy scikit-learn scipy statsmodels --upgrade :: python3.7 -c "import imageio;imageio.plugins.ffmpeg.download()" For :epkg:`spacy`: :: pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fr_core_news_sm-2.0.0-py3-none-any.whl For module :epkg:`imageio`, the second instruction must be executed with every distinct user once: Install :epkg:`clang` 6 (for :epkg:`pythonnet`): :: echo "deb http://ftp.debian.org/debian stretch-backports main" | sudo tee /etc/apt/sources.list.d/clang-stable.list apt-get update apt-get install -y clang clang-6.0 clang-3.8 alias clang='clang-6.0' Install :epkg:`mono` (for :epkg:`pythonnet`, skip?): :: apt-get install -y apt-transport-https apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list apt-get update apt-get install -y mono-devel ca-certificates-mono And for :epkg:`dotnet` (see `Install .NET Core SDK on Linux Debian 9 `_): :: wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/ wget -q https://packages.microsoft.com/config/debian/9/prod.list mv prod.list /etc/apt/sources.list.d/microsoft-prod.list chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg chown root:root /etc/apt/sources.list.d/microsoft-prod.list apt-get update apt-get install -y dotnet-sdk-2.2 Install :epkg:`pythonnet`: :: pip3.7 install git+https://github.com/rkern/line_profiler.git git clone -b master https://github.com/pythonnet/pythonnet.git cd pythonnet python3.7 setup.py install --xplat Deal with permissions issues for nugets (wait for some nuget to be installed): :: chown -R jenkins:jenkins /tmp/NuGetScratch/ For :epkg:`ML.net` (see `Building ML.NET on Linux and macOS `_): :: apt-get update You might need to run ``chmod 777 -R /tmp/NuGetScratch;chmod 777 -R /tmp/NuGet``. I got the following issue `Restore task fails in an multi user scenario `_. For :epkg:`onnx`: :: pip3.7 install onnx **MKL** and **deep learning** The machine I used does not have GPU. Missing :epkg:`MKL` : `MKL for .deb-based systems: An easy recipe `_. About :epkg:`pyopencl` ... :: apt-get install -y beignet-opencl-icd ocl-icd-opencl-dev libpocl-dev nvidia-cuda-toolkit pip3.7 install pyopencl pycuda tf-nightly azure To reboot: :: reboot Stores password with :epkg:`keyring`. First switch to :epkg:`Jenkins` user: :: su su - jenkins Then: :: import keyring import os keyring.set_password("web", "pyquickhelper,user", "...") keyring.set_password("web", "pyquickhelper,pwd", "...") mkdir -p somewhere/workspace/local_pypi/local_pypi_server About *Jenkins*: * `How do I change the Jenkins service account user? `_ A few useful lines maybe: :: nano /etc/init.d/jenkins chown -R dupre:dupre somewhere chown -R dupre:dupre /var/cache/jenkins chown -R dupre:dupre /var/log/jenkins chown -R jenkins:jenkins somewhere chown -R jenkins:jenkins /var/cache/jenkins chown -R jenkins:jenkins /var/log/jenkins /etc/init.d/jenkins restart For :epkg:`spacy`, to download the ressources for a particular language: :: python3.7 -m spacy download en Server FTP, look into `How to Configure VSFTPD FTPS with SSL/TLS on Ubuntu 18.04 `_.