pip install pyquickhelper
If you want to contribute, you need to fork and clone this reposity sdpython/pyquickhelper. Otherwise, a zip file of the sources is enough.
git clone https://github.com/sdpython/pyquickhelper.git cd pyquickhelper python setup.py install
To generate a zip or gz setup:
python setup.py sdist --formats=gztar,zip
To generate a file *.whl:
python setup.py bdist_wheel
The setup implements other commands to help writing code or testing notebooks.
from pyquickhelper.pycode.setup_helper import get_available_setup_commands print("\n".join(sorted(get_available_setup_commands())))
bdist_egg bdist_msi bdist_wheel bdist_wininst build27 build_ext build_script build_sphinx clean_pyd clean_space copy27 copy_dist copy_sphinx history lab local_jenkins local_pypi notebook publish publish_doc register run27 run_pylint sdist setupdep test_local_pypi unittests unittests_GUI unittests_LONG unittests_SKIP upload_docs write_version
The most used one is
python setup.py clean_space. The commands
modifies the files to be closer to pep8 conventions, it also
runs some checkings to display warnings for remaining issues.
python setup.py run_pylint runs pylint
on the code. Extended syntax is
python setup.py run_pylint <filter> <negative filte> -iXXXX.
-iXXXX ignores warnings XXXX. One must be added for each
warning to ignore.
python setup.py buld_sphinx runs the
python setup.py copy27,
python setup.py run27,
python setup.py build27 converts the module for Python 2.7
(not bullet proof),
python setup.py history retrieves the
history and update file
HISTORY.rst on root folder.
python setup.py notebook starts a local Jupyter
notebook server to easily modify the notebooks,
python setup.py lab
does the same for Jupyter Lab. Finally,
a script to start SciTe with the right path
on Windows (to be saved in
@echo off set MYPYTHON=C:\Python36_x64 set PYADDPATH=C:\username\GitHub set PATH=%MYPYTHON%;%PATH% set PYTHONPATH=%PYADDPATH%\pyquickhelper\src;%PYADDPATH%\pyensae\src;%PYADDPATH%\pyrsslocal\src;%PYADDPATH%\jyquickhelper\src set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\manydataapi\src;%PYADDPATH%\ensae_teaching_cs\src;%PYADDPATH%\mlinsights\src;%PYADDPATH%\mlstatpy\src set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\actuariat_python\src;%PYADDPATH%\code_beatrix\src;%PYADDPATH%\cpyquickhelper\src;%PYADDPATH%\ensae_projects\src set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\lightmlrestapi\src;%PYADDPATH%\lightmlboard\src;%PYADDPATH%\pandas_streaming\src;%PYADDPATH%\papierstat\src set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\pymmails\src;%PYADDPATH%\pysqllike\src;%PYADDPATH%\pymyinstall\src;%PYADDPATH%\teachpyx\src start /b pathtoscite\SciTE.exe
It relies on pyquickhelper.
You need to get the sources and run:
python -u setup.py unittests
There are more options.
[-d seconds]: run all unit tests for which predicted duration is below a given threshold.
[-f file]: run all unit tests in file (do not use the full path)
[-e regex]: run all unit tests files matching the regular expression (can be combined with
[-g regex]: run all unit tests files not matching the regular expression
You can get them with:
python setup.py unittests --help
The command line runs the unit tests.
The process ends with the code coverage (with module coverage)
and publishes the report in folder _doc/sphinxdoc/source/coverage.
-g are left empty, files containing test_LONG_,
test_SKIP_, test_GUI_ in their
name are included. You can run them with a specific command:
python setup.py unittests_LONG python setup.py unittests_SKIP python setup.py unittests_GUI
This was introduced to explicitely exclude long tests used to check a long process was not broken. These commands do not accept parameters. Coverage reports are not merged.
All unit tests must follow the convention:
This test file must begin by test_ and must look like the following:
""" @file @brief test log(time=2s) """ import sys import os import unittest from pyquickhelper import fLOG # import the file you want to test from project_name.subproject.myexample import myclass class TestExample(unittest.TestCase): def test_split_cmp_command(self) : # to log information only when run as main file fLOG (__file__, self._testMethodName, OutputPrint = __name__ == "__main__") # you test content # it must raises an exception if a test fails. if __name__ == "__main__" : unittest.main ()
You can check if the test is run on a specific environment:
from pyquickhelper.pycode import skipif_travis, skipif_circleci from pyquickhelper.pycode import skipif_appveyor, skipif_azure
Function is_travis_or_appveyor return a string
'appveyor' is the code is executed on such environment or None if
none of them is detected.
You can create a temporary folder next to the test file by running:
from pyquickhelper.pycode import get_temp_folder temp = get_temp_folder(__file__, "temp_<name>")
This folder is automatically removed if it exists when the function is called.
The unit test test_flake8.py ensures all the code follows the pep8 style. It will break it is not the case and will indicate where it breaks. The code can be automatically modified to follow that convention by running:
python setup.py clean_space
The unit test test_convert_notebooks.py checks the syntax of every notebook looks ok. This tests also removes all execution number and reformat the JSON. It must be run before a commit if you add or modifies the notebook. Removing the execution number makes it easy to compare two versions of the same notebook.
Notebooks are not tested by default but they should wherever it is possible.
This can be done by using function
If a couple of unit test requires a login and a password to test FTP functionalities for example, you should get them with keyring.
import keyring keyring.get_password("something", os.environ["COMPUTERNAME"] + "user")
You can set the password by running only once:
import keyring keyring.set_password("something", os.environ["COMPUTERNAME"] + "user", "...") keyring.set_password("something", os.environ["COMPUTERNAME"] + "pwd", "...")
The sources can not be used with Python 2.7. The syntax first needs to be converted. This is what the following instruction based on Python 3 does. The results will located in dist_module27.
python setup.py copy27
From folder dist_module27, the unit test can be run he same way:
python setup.py unittests
Or with nose:
nosetests.exe -w ut_<folder_name>
It relies on epkg:pyquickhelper.
python setup.py build_sphinx
It requires the full sources from GitHub and not only the installed package which does not contains the documentation. It will go through the following steps:
It will copy all files found in src in folder _doc/sphinxdoc/source/[project_name].
It will generates a file .rst for each python file in _doc/sphinxdoc/source/[project_name].
It will run the generation of the documentation using Sphinx.
Notebooks can be placed in _doc/notebooks, they will be added to the documentation.
It will generated aggregated pages for blog posts added to _doc/sphinxdoc/source/blog/YYYY/<anything>.rst.
The results are stored in folder _doc/sphinxdoc/build. The process requires dependencies:
Required sphinx extensions can be found in the code of set_sphinx_variables <pyquickhelper.helpgen.default_conf.set_sphinx_variables>
As the documentation creates graphs to represent the dependencies, Graphviz needs to be installed. Here is the list of required tools:
If you need to use Antlr:
Startup Trigger: automatisation de build
# -*- coding: utf-8 -*- """ @file @brief Configuration for sphinx documentation. """ import sys import os import alabaster from sphinx.builders.html import Stylesheet sys.path.insert(0, os.path.abspath(os.path.join(os.path.split(__file__)))) from pyquickhelper.helpgen.default_conf import set_sphinx_variables def callback_begin(): print("[sphinx.conf.py] documentation started.") set_sphinx_variables(__file__, "pyquickhelper", "Xavier Dupré", 2022, "alabaster", alabaster.get_path(), locals(), github_repo="https://github.com/sdpython/pyquickhelper.git", extlinks=dict(issue=( 'https://github.com/sdpython/pyquickhelper/issues/%s', 'issue ')), link_resolve="http://www.xavierdupre.fr/app/", callback_begin=callback_begin) # there is an issue with this attribute on Anaconda math_number_all assert math_number_all or not math_number_all blog_root = "http://www.xavierdupre.fr/app/pyquickhelper/helpsphinx/" # remove notebooks following this pattern nbneg_pattern = ".*[\\\\/]temp_.*" html_css_files = ['my-styles.css', 'gallery-dataframe.css']
The documentation is organized as follows:
src/<module_name>: contains the sources of the modules
_doc/notebooks: contains the notebooks included in the documentation
_doc/sphinxdoc/source: contains the sphinx documentation
_doc/sphinxdoc/blog/YYYY: contains the blog posts for year YYYY
When the documentation is being generated, the sources are copied into pyquickhelper/_unittests/_doc/sphinxdoc/source/pyquickhelper. The documentation can be in javadoc format is replaced by the RST syntax. Various files are automatically generated (indexes, examples, FAQ). Then sphinx is run.
You will find some examples of custom sphinx commands in Examples for the documentation.
List of Sphinx commands added by pyquickhelper
bigger: to write with a custom size
blocref: to add a definition (or any kind of definition)
blocreflist: to list all definitions
blogpost: to add a blog post, this command does not behave like the others, it should only be used in folder _doc/sphinxdoc/source/blog
blogpostagg: to aggregate blog post, this should be manually added, the module pyquickhelper is preprocessing the documentation to produce pages containing such commands
cmdref: to documentation a script the module makes available on the command line
cmdreflist: to list all commands
epkg: avoid repeating the same references in many places
exref: to add an example
exreflist: to list all example
faqref: to add a FAQ
faqreflist: to list all FAQ
mathdef: to add a mathematical definition (or any kind of definition)
mathdeflist: to list all definitions
nbref: to add a magic command
nbreflist: to list all magic commands
runpython: to run a script and display the output, it can be used to generate documentation
sharenet: to add buttons to share the page on a socal network
todoext: to add an issue or a work item
todoextlist: to list all issues or work item
These commands are documented in Sphinx Extensions.
Notebooks in folder _doc/notebooks will be automatically converted into html, rst, pdf, slides formats. That requires latex and pandoc.
The module is tested with Travis, AppVeyor and local testing with Jenkins for a exhaustive list of unit tests, the documentation, the setup. Everything is fully tested on Windows with the standard distribution and Anaconda. There are three builds definition:
The third file by processed by pyquickhelper itself to produce a series of Jenkins jobs
uploaded to a server. See
to configurate a local Jenkins server.
When modules depend on others modules also being tested, the unit tests and the documentation generation uses a local pypi server (port=8079).
The plugin Collapsing Sections Plugins can help parsing the output. The following section are added:
---- JENKINS BEGIN UNIT TESTS ----, *
---- JENKINS END UNIT TESTS ----
---- JENKINS BEGIN DOCUMENTATION ----,
---- JENKINS END DOCUMENTATION ----
---- JENKINS BEGIN DOCUMENTATION NOTEBOOKS ----,
---- JENKINS END DOCUMENTATION NOTEBOOKS ----
---- JENKINS BEGIN DOCUMENTATION BLOG ----,
---- JENKINS END DOCUMENTATION BLOG ----
---- JENKINS BEGIN DOCUMENTATION COPY FILES ----,
---- JENKINS END DOCUMENTATION COPY FILES ----
---- JENKINS BEGIN DOCUMENTATION ENCODING ----,
---- JENKINS END DOCUMENTATION ENCODING ----
---- JENKINS BEGIN DOCUMENTATION SPHINX ----,
---- JENKINS END DOCUMENTATION SPHINX ----
---- JENKINS BEGIN WRITE VERSION ----,
---- JENKINS END WRITE VERSION ----
---- JENKINS BEGIN SETUPHOOK ----,
---- JENKINS END SETUPHOOK ----