Contribute

Installation

Installation with pip

pip install pyquickhelper

Installation with the source

If you want to contribute, you need to fork and clone this reposity sdpython/pyquickhelper. Otherwise, a zip file of the sources is enough.

git clone https://github.com/sdpython/pyquickhelper.git
cd pyquickhelper
python setup.py install

Generate the setup

Build the wheel

To generate a zip or gz setup:

python setup.py sdist --formats=gztar,zip

To generate a file *.whl:

python setup.py bdist_wheel

Other available commands

The setup implements other commands to help writing code or testing notebooks.

<<<

from pyquickhelper.pycode.setup_helper import get_available_setup_commands
print("\n".join(sorted(get_available_setup_commands())))

>>>

    bdist_egg
    bdist_msi
    bdist_wheel
    bdist_wininst
    build27
    build_ext
    build_script
    build_sphinx
    clean_pyd
    clean_space
    copy27
    copy_dist
    copy_sphinx
    history
    lab
    local_jenkins
    local_pypi
    notebook
    publish
    publish_doc
    register
    run27
    run_pylint
    sdist
    setup_hook
    setupdep
    test_local_pypi
    unittests
    unittests_GUI
    unittests_LONG
    unittests_SKIP
    upload_docs
    write_version

The most used one is python setup.py clean_space. The commands modifies the files to be closer to pep8 conventions, it also runs some checkings to display warnings for remaining issues. The command python setup.py run_pylint runs pylint on the code. Extended syntax is python setup.py run_pylint <filter> <negative filte> -iXXXX. -iXXXX ignores warnings XXXX. One must be added for each warning to ignore. python setup.py buld_sphinx runs the documentation, python setup.py copy27, python setup.py run27, python setup.py build27 converts the module for Python 2.7 (not bullet proof), python setup.py history retrieves the history and update file HISTORY.rst on root folder.

python setup.py notebook starts a local Jupyter notebook server to easily modify the notebooks, python setup.py lab does the same for Jupyter Lab. Finally, a script to start SciTe with the right path on Windows (to be saved in something.bat).

@echo off

set MYPYTHON=C:\Python36_x64
set PYADDPATH=C:\username\GitHub
set PATH=%MYPYTHON%;%PATH%
set PYTHONPATH=%PYADDPATH%\pyquickhelper\src;%PYADDPATH%\pyensae\src;%PYADDPATH%\pyrsslocal\src;%PYADDPATH%\jyquickhelper\src
set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\manydataapi\src;%PYADDPATH%\ensae_teaching_cs\src;%PYADDPATH%\mlinsights\src;%PYADDPATH%\mlstatpy\src
set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\actuariat_python\src;%PYADDPATH%\code_beatrix\src;%PYADDPATH%\cpyquickhelper\src;%PYADDPATH%\ensae_projects\src
set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\lightmlrestapi\src;%PYADDPATH%\lightmlboard\src;%PYADDPATH%\pandas_streaming\src;%PYADDPATH%\papierstat\src
set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\pymmails\src;%PYADDPATH%\pysqllike\src;%PYADDPATH%\pymyinstall\src;%PYADDPATH%\teachpyx\src
start /b pathtoscite\SciTE.exe

Unit tests

It relies on pyquickhelper.

Run unit tests

You need to get the sources and run:

python -u setup.py unittests

There are more options.

  • [-d seconds]: run all unit tests for which predicted duration is below a given threshold.
  • [-f file]: run all unit tests in file (do not use the full path)
  • [-e regex]: run all unit tests files matching the regular expression (can be combined with -g)
  • [-g regex]: run all unit tests files not matching the regular expression

You can get them with:

python setup.py unittests --help

The process first calls the function _setup_hook. It can be used to initialize the module even if it is most of the time unused. The process ends the code coverage (with module coverage) and publishes the report in folder _doc/sphinxdoc/source/coverage. If options -e and -g are left empty, files containing test_LONG_, test_SKIP_, test_GUI_ in their name are included. You can run them with a specific command:

python setup.py unittests_LONG
python setup.py unittests_SKIP
python setup.py unittests_GUI

This was introduced to explicitely exclude long tests used to check a long process was not broken. These commands do not accept parameters. Coverage reports are not merged.

Write and run one unit test

All unit tests must follow the convention:

_unittests/ut_<subfolder>/test_<filename>.py

This test file must begin by test_ and must look like the following:

"""
@file
@brief  test log(time=2s)
"""
import sys
import os
import unittest

# to import files from the module
# and to make sure we do not use another installed version
try :
    import src
except ImportError :
    path = os.path.normpath(os.path.abspath( os.path.join( os.path.split(__file__)[0], "..", "..")))
    if path not in sys.path : sys.path.append (path)
    import src

from pyquickhelper import fLOG

# import the file you want to test
from src.project_name.subproject.myexample import myclass

class TestExample (unittest.TestCase):

    def test_split_cmp_command(self) :

        # to log information
        fLOG (__file__, self._testMethodName, OutputPrint = __name__ == "__main__")

        # you test content
        # it must raises an exception if a test fails.

if __name__ == "__main__"  :
    unittest.main ()

You can check if the test is run on a specific environment:

from pyquickhelper.pycode import is_travis_or_appveyor

Function is_travis_or_appveyor return a string 'travis' or 'appveyor' is the code is executed on such environment or None if none of them is detected. You can create a temporary folder next to the test file by running:

from pyquickhelper.pycode import get_temp_folder
temp = get_temp_folder(__file__, "temp_<name>")

This folder is automatically removed if it exists when the function is called.

Specific unit tests

The unit test test_flake8.py ensures all the code follows the pep8 style. It will break it is not the case and will indicate where it breaks. The code can be automatically modified to follow that convention by running:

python setup.py clean_space

The unit test test_readme.py checks the syntax of file readme.rst. PyPi runs on an older version of docutils. It checks this file follows this syntax.

The unit test test_convert_notebooks.py checks the syntax of every notebook looks ok. This tests also removes all execution number and reformat the JSON. It must be run before a commit if you add or modifies the notebook. Removing the execution number makes it easy to compare two versions of the same notebook.

Notebooks are not tested by default but they should wherever it is possible. This can be done by using function execute_notebook_list.

Use passwords

If a couple of unit test requires a login and a password to test FTP functionalities for example, you should get them with keyring.

import keyring
keyring.get_password("something", os.environ["COMPUTERNAME"] + "user")

You can set the password by running only once:

import keyring
keyring.set_password("something", os.environ["COMPUTERNAME"] + "user", "...")
keyring.set_password("something", os.environ["COMPUTERNAME"] + "pwd", "...")

Python 2.7

The sources can not be used with Python 2.7. The syntax first needs to be converted. This is what the following instruction based on Python 3 does. The results will located in dist_module27.

python setup.py copy27

From folder dist_module27, the unit test can be run he same way:

python setup.py unittests

Or with nose:

nosetests.exe -w ut_<folder_name>

Documentation

It relies on epkg:pyquickhelper.

Generation

The documentation can be written using RST format or javadoc format. The documentation can generated by:

python setup.py build_sphinx

It requires the full sources from GitHub and not only the installed package which does not contains the documentation. It will go through the following steps:

  • It gets a version number from git (the sources must be on git).
  • It will copy all files found in src in folder _doc/sphinxdoc/source/[project_name].
  • It will generates a file .rst for each python file in _doc/sphinxdoc/source/[project_name].
  • It will run the generation of the documentation using Sphinx.
  • Notebooks can be placed in _doc/notebooks, they will be added to the documentation.
  • It will generated aggregated pages for blog posts added to _doc/sphinxdoc/source/blog/YYYY/<anything>.rst.

The results are stored in folder _doc/sphinxdoc/build. The process requires dependencies:

  • Sphinx
  • pyquickhelper
  • Required sphinx extensions can be found in the code of set_sphinx_variables <pyquickhelper.helpgen.default_conf.set_sphinx_variables>

As the documentation creates graphs to represent the dependencies, Graphviz needs to be installed. Here is the list of required tools:

If you need to use Antlr:

Jenkins extensions:

The module will convert SVG into images,
it can handle javascript with module js2py and node.js.

Configuration

# -*- coding: utf-8 -*-
"""
@file
@brief Configuration for sphinx documentation.
"""
import sys
import os
import sphinx_rtd_theme

sys.path.insert(0, os.path.abspath(os.path.join(os.path.split(__file__)[0])))
from pyquickhelper.helpgen.default_conf import set_sphinx_variables, get_default_stylesheet

set_sphinx_variables(__file__, "pyquickhelper", "Xavier Dupré", 2019,
                     "sphinx_rtd_theme", [
                         sphinx_rtd_theme.get_html_theme_path()], locals(),
                     github_repo="https://github.com/sdpython/pyquickhelper.git",
                     extlinks=dict(issue=(
                         'https://github.com/sdpython/pyquickhelper/issues/%s',
                         'issue ')),
                     link_resolve="http://www.xavierdupre.fr/app/")

# there is an issue with this attribute on Anaconda math_number_all
assert math_number_all or not math_number_all
blog_root = "http://www.xavierdupre.fr/app/pyquickhelper/helpsphinx/"

# remove notebooks following this pattern
nbneg_pattern = ".*[\\\\/]temp_.*"

html_context = {
    'css_files': get_default_stylesheet() + ['_static/my-styles.css', '_static/gallery.css'],
}

Write documentation

The documentation is organized as follows:

  • src/<module_name>: contains the sources of the modules
  • _doc/notebooks: contains the notebooks included in the documentation
  • _doc/sphinxdoc/source: contains the sphinx documentation
  • _doc/sphinxdoc/blog/YYYY: contains the blog posts for year YYYY

When the documentation is being generated, the sources are copied into pyquickhelper/_unittests/_doc/sphinxdoc/source/pyquickhelper. The documentation can be in javadoc format is replaced by the RST syntax. Various files are automatically generated (indexes, examples, FAQ). Then sphinx is run.

You will find some examples of custom sphinx commands in Examples for the documentation.

List of Sphinx commands added by pyquickhelper

  • bigger: to write with a custom size
  • blocref: to add a definition (or any kind of definition)
  • blocreflist: to list all definitions
  • blogpost: to add a blog post, this command does not behave like the others, it should only be used in folder _doc/sphinxdoc/source/blog
  • blogpostagg: to aggregate blog post, this should be manually added, the module pyquickhelper is preprocessing the documentation to produce pages containing such commands
  • cmdref: to documentation a script the module makes available on the command line
  • cmdreflist: to list all commands
  • epkg: avoid repeating the same references in many places
  • exref: to add an example
  • exreflist: to list all example
  • faqref: to add a FAQ
  • faqreflist: to list all FAQ
  • mathdef: to add a mathematical definition (or any kind of definition)
  • mathdeflist: to list all definitions
  • nbref: to add a magic command
  • nbreflist: to list all magic commands
  • runpython: to run a script and display the output, it can be used to generate documentation
  • sharenet: to add buttons to share the page on a socal network
  • todoext: to add an issue or a work item
  • todoextlist: to list all issues or work item

These commands are documented in Sphinx Extensions.

Notebooks

Notebooks in folder _doc/notebooks will be automatically converted into html, rst, pdf, slides formats. That requires latex and pandoc.

Continuous Integration

The module is tested with Travis, AppVeyor and local testing with Jenkins for a exhaustive list of unit tests, the documentation, the setup. Everything is fully tested on Windows with the standard distribution and Anaconda. There are three builds definition:

The third file by processed by pyquickhelper itself to produce a series of Jenkins jobs uploaded to a server. See setup_jenkins_server_yml to configurate a local Jenkins server.

When modules depend on others modules also being tested, the unit tests and the documentation generation uses a local pypi server (port=8079).

Console Output

The plugin Collapsing Sections Plugins can help parsing the output. The following section are added:

  • ---- JENKINS BEGIN UNIT TESTS ----, * ---- JENKINS END UNIT TESTS ----
  • ---- JENKINS BEGIN DOCUMENTATION ----, ---- JENKINS END DOCUMENTATION ----
  • ---- JENKINS BEGIN DOCUMENTATION NOTEBOOKS ----, ---- JENKINS END DOCUMENTATION NOTEBOOKS ----
  • ---- JENKINS BEGIN DOCUMENTATION BLOG ----, ---- JENKINS END DOCUMENTATION BLOG ----
  • ---- JENKINS BEGIN DOCUMENTATION COPY FILES ----, ---- JENKINS END DOCUMENTATION COPY FILES ----
  • ---- JENKINS BEGIN DOCUMENTATION ENCODING ----, ---- JENKINS END DOCUMENTATION ENCODING ----
  • ---- JENKINS BEGIN DOCUMENTATION SPHINX ----, ---- JENKINS END DOCUMENTATION SPHINX ----
  • ---- JENKINS BEGIN WRITE VERSION ----, ---- JENKINS END WRITE VERSION ----
  • ---- JENKINS BEGIN SETUPHOOK ----, ---- JENKINS END SETUPHOOK ----