2017-08-28 Continuous integration
Maintaining notebooks is a lot of work. I consider a notebook is ok if it runs with the latest versions of the packages it uses and if it can be converted into HTML to be included on a website. Easy when you have one notebook but what if you have more than 200 hundred of them to verify for your teachings? I had to automate. I started by running a virtual machine on Azure wich Jenkins and everything I needed. This machine is still up and running and tests everything once in a week for all my python package. But my teachings are also open source so I decided to use continuous integration to test my package on other distributions. I first used travis and appveyor but I could not include the compilation of the notebooks into documentation. It requires a couple of huge dependencies as latex. appveyor is quite slow and stops any job after one hour. travis was using Ubuntu 12.04 when I started and 14.04 now. I recently tried CircleCI 2.0. The design is really nice, it offers Ubuntu 16.04, the configuration file config.yml is much easier to read and to write. One interesting feature is artifacts. The user can easily copy in a specific folder whatever he wants to keep from the build and make it available to download. The other interesting feature is caching. CircleCI automatically caches a docker which contains the dependencies of the package to test. The first run is slow and the following are faster due to this option plus renewing the cache can be easily done by changing a file. So I decided it was worth spending some time enabling CircleCI anywhere. I described here all the steps to create a Python package, with unit tests and documentation and to build it on CircleCI : Tests unitaires, setup et ingéniérie logiciel (French).
2015-10-27 Travis, Appveyor, PyPi
I was surprised to see that a module I develop to produce my teaching materials gets downloaded 14k times last month on PiPy (pyquickhelper). It seems a lot... But if I count that Travis downloads it everytime I commit something on GitHub. I did 50 commits last week. I would say that the number of time I manually download this module is not significant compare to the number of times it gets automatically downloaded. It seems difficult to get a sense of those counts given by PiPy.
2015-05-19 Continuous integration
I'm using Travis to check that my open source modules works on Linux. I discovered the following video Olivier Grisel - Build and test wheel packages on Linux, OSX & Windows - PyCon 2015 which explains how this is done for scikit-learn. AppVeyor offers the same service as Travis but for Windows (it works with Azure). The setup is more complex as Python is the C++ compiler is still a pain to configure. You can check how it is done for scikit-learn: windows_testing_downloader.ps1, appveyor.yml. A few other interesting scripts: mingw.py.
2015-02-28 Automated build on Travis for a python module
Many python modules display a small logo
which indicates the build status:
.
I set up the same for the module
pyquickhelper which is held on github/pyquickhelper.
Travis installs packages before building the modules.
The first step is to gather all the dependencies:
pip freeze > requirements.txt
I replaced == by >= and removed some of them, I got:
Cython>=0.20.2 Flask>=0.10.1 Flask-SQLAlchemy>=2.0 Jinja2>=2.7.3 Markdown>=2.4.1 ...
more...
|
Xavier Dupré
|
