Contribute¶
Installation¶
Installation with pip¶
pip install pyquickhelper
Installation with the source¶
If you want to contribute, you need to fork and clone this reposity sdpython/pyquickhelper. Otherwise, a zip file of the sources is enough.
git clone https://github.com/sdpython/pyquickhelper.git
cd pyquickhelper
python setup.py install
Generate the setup¶
Build the wheel¶
To generate a zip or gz setup:
python setup.py sdist --formats=gztar,zip
To generate a file *.whl:
python setup.py bdist_wheel
Other available commands¶
The setup implements other commands to help writing code or testing notebooks.
<<<
from pyquickhelper.pycode.setup_helper import get_available_setup_commands
print("\n".join(sorted(get_available_setup_commands())))
>>>
bdist_egg
bdist_msi
bdist_wheel
bdist_wininst
build27
build_ext
build_script
build_sphinx
clean_pyd
clean_space
copy27
copy_dist
copy_sphinx
history
lab
local_jenkins
local_pypi
notebook
publish
publish_doc
register
run27
run_pylint
sdist
setupdep
test_local_pypi
unittests
unittests_GUI
unittests_LONG
unittests_SKIP
upload_docs
write_version
The most used one is python setup.py clean_space
. The commands
modifies the files to be closer to pep8 conventions, it also
runs some checkings to display warnings for remaining issues.
The command python setup.py run_pylint
runs pylint
on the code. Extended syntax is
python setup.py run_pylint <filter> <negative filte> -iXXXX
.
-iXXXX
ignores warnings XXXX. One must be added for each
warning to ignore. python setup.py buld_sphinx
runs the
documentation, python setup.py copy27
, python setup.py run27
,
python setup.py build27
converts the module for Python 2.7
(not bullet proof), python setup.py history
retrieves the
history and update file HISTORY.rst
on root folder.
python setup.py notebook
starts a local Jupyter
notebook server to easily modify the notebooks, python setup.py lab
does the same for Jupyter Lab. Finally,
a script to start SciTe with the right path
on Windows (to be saved in something.bat
).
@echo off
set MYPYTHON=C:\Python36_x64
set PYADDPATH=C:\username\GitHub
set PATH=%MYPYTHON%;%PATH%
set PYTHONPATH=%PYADDPATH%\pyquickhelper\src;%PYADDPATH%\pyensae\src;%PYADDPATH%\pyrsslocal\src;%PYADDPATH%\jyquickhelper\src
set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\manydataapi\src;%PYADDPATH%\ensae_teaching_cs\src;%PYADDPATH%\mlinsights\src;%PYADDPATH%\mlstatpy\src
set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\actuariat_python\src;%PYADDPATH%\code_beatrix\src;%PYADDPATH%\cpyquickhelper\src;%PYADDPATH%\ensae_projects\src
set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\lightmlrestapi\src;%PYADDPATH%\lightmlboard\src;%PYADDPATH%\pandas_streaming\src;%PYADDPATH%\papierstat\src
set PYTHONPATH=%PYTHONPATH%;%PYADDPATH%\pymmails\src;%PYADDPATH%\pysqllike\src;%PYADDPATH%\pymyinstall\src;%PYADDPATH%\teachpyx\src
start /b pathtoscite\SciTE.exe
Unit tests¶
It relies on pyquickhelper.
Run unit tests¶
You need to get the sources and run:
python -u setup.py unittests
There are more options.
[-d seconds]
: run all unit tests for which predicted duration is below a given threshold.[-f file]
: run all unit tests in file (do not use the full path)[-e regex]
: run all unit tests files matching the regular expression (can be combined with-g
)[-g regex]
: run all unit tests files not matching the regular expression
You can get them with:
python setup.py unittests --help
The command line runs the unit tests.
The process ends with the code coverage (with module coverage)
and publishes the report in folder _doc/sphinxdoc/source/coverage.
If options -e
and -g
are left empty, files containing test_LONG_,
test_SKIP_, test_GUI_ in their
name are included. You can run them with a specific command:
python setup.py unittests_LONG
python setup.py unittests_SKIP
python setup.py unittests_GUI
This was introduced to explicitely exclude long tests used to check a long process was not broken. These commands do not accept parameters. Coverage reports are not merged.
Write and run one unit test¶
All unit tests must follow the convention:
_unittests/ut_<subfolder>/test_<filename>.py
This test file must begin by test_ and must look like the following:
"""
@file
@brief test log(time=2s)
"""
import sys
import os
import unittest
from pyquickhelper import fLOG
# import the file you want to test
from project_name.subproject.myexample import myclass
class TestExample(unittest.TestCase):
def test_split_cmp_command(self) :
# to log information only when run as main file
fLOG (__file__, self._testMethodName, OutputPrint = __name__ == "__main__")
# you test content
# it must raises an exception if a test fails.
if __name__ == "__main__" :
unittest.main ()
You can check if the test is run on a specific environment:
from pyquickhelper.pycode import skipif_travis, skipif_circleci
from pyquickhelper.pycode import skipif_appveyor, skipif_azure
Function is_travis_or_appveyor return a string
'travis'
or 'appveyor'
is the code is executed on such environment or None if
none of them is detected.
You can create a temporary folder next to the test file by running:
from pyquickhelper.pycode import get_temp_folder
temp = get_temp_folder(__file__, "temp_<name>")
This folder is automatically removed if it exists when the function is called.
Specific unit tests¶
The unit test test_flake8.py ensures all the code follows the pep8 style. It will break it is not the case and will indicate where it breaks. The code can be automatically modified to follow that convention by running:
python setup.py clean_space
The unit test test_readme.py checks the syntax of file readme.rst. PyPi runs on an older version of docutils. It checks this file follows this syntax.
The unit test test_convert_notebooks.py checks the syntax of every notebook looks ok. This tests also removes all execution number and reformat the JSON. It must be run before a commit if you add or modifies the notebook. Removing the execution number makes it easy to compare two versions of the same notebook.
Notebooks are not tested by default but they should wherever it is possible.
This can be done by using function
execute_notebook_list
.
Use passwords¶
If a couple of unit test requires a login and a password to test FTP functionalities for example, you should get them with keyring.
import keyring
keyring.get_password("something", os.environ["COMPUTERNAME"] + "user")
You can set the password by running only once:
import keyring
keyring.set_password("something", os.environ["COMPUTERNAME"] + "user", "...")
keyring.set_password("something", os.environ["COMPUTERNAME"] + "pwd", "...")
Python 2.7¶
The sources can not be used with Python 2.7. The syntax first needs to be converted. This is what the following instruction based on Python 3 does. The results will located in dist_module27.
python setup.py copy27
From folder dist_module27, the unit test can be run he same way:
python setup.py unittests
Or with nose:
nosetests.exe -w ut_<folder_name>
Documentation¶
It relies on epkg:pyquickhelper.
Generation¶
The documentation can be written using RST format or javadoc format. The documentation can generated by:
python setup.py build_sphinx
It requires the full sources from GitHub and not only the installed package which does not contains the documentation. It will go through the following steps:
It gets a version number from git (the sources must be on git).
It will copy all files found in src in folder _doc/sphinxdoc/source/[project_name].
It will generates a file .rst for each python file in _doc/sphinxdoc/source/[project_name].
It will run the generation of the documentation using Sphinx.
Notebooks can be placed in _doc/notebooks, they will be added to the documentation.
It will generated aggregated pages for blog posts added to _doc/sphinxdoc/source/blog/YYYY/<anything>.rst.
The results are stored in folder _doc/sphinxdoc/build. The process requires dependencies:
Required sphinx extensions can be found in the code of set_sphinx_variables <pyquickhelper.helpgen.default_conf.set_sphinx_variables>
As the documentation creates graphs to represent the dependencies, Graphviz needs to be installed. Here is the list of required tools:
If you need to use Antlr:
Jenkins extensions:
Startup Trigger: automatisation de build
- The module will convert SVG into images,
it can handle javascript with module js2py and node.js.
Configuration¶
# -*- coding: utf-8 -*-
"""
@file
@brief Configuration for sphinx documentation.
"""
import sys
import os
import alabaster
import bokeh.sphinxext.bokeh_plot
sys.path.insert(0, os.path.abspath(os.path.join(os.path.split(__file__)[0])))
from pyquickhelper.helpgen.default_conf import set_sphinx_variables
set_sphinx_variables(__file__, "pyquickhelper", "Xavier Dupré", 2023,
"alabaster", alabaster.get_path(),
locals(),
github_repo="https://github.com/sdpython/pyquickhelper.git",
extlinks=dict(issue=(
'https://github.com/sdpython/pyquickhelper/issues/%s',
'issue %s')),
link_resolve="http://www.xavierdupre.fr/app/")
extensions.extend([
"bokeh.sphinxext.bokeh_plot",
])
# there is an issue with this attribute on Anaconda math_number_all
assert math_number_all or not math_number_all
blog_root = "http://www.xavierdupre.fr/app/pyquickhelper/helpsphinx/"
# remove notebooks following this pattern
nbneg_pattern = ".*[\\\\/]temp_.*"
html_css_files = ['my-styles.css', 'gallery-dataframe.css']
Write documentation¶
The documentation is organized as follows:
src/<module_name>: contains the sources of the modules
_doc/notebooks: contains the notebooks included in the documentation
_doc/sphinxdoc/source: contains the sphinx documentation
_doc/sphinxdoc/blog/YYYY: contains the blog posts for year YYYY
When the documentation is being generated, the sources are copied into pyquickhelper/_unittests/_doc/sphinxdoc/source/pyquickhelper. The documentation can be in javadoc format is replaced by the RST syntax. Various files are automatically generated (indexes, examples, FAQ). Then sphinx is run.
You will find some examples of custom sphinx commands in Examples for the documentation.
List of Sphinx commands added by pyquickhelper
bigger
: to write with a custom sizeblocref
: to add a definition (or any kind of definition)blocreflist
: to list all definitionsblogpost
: to add a blog post, this command does not behave like the others, it should only be used in folder _doc/sphinxdoc/source/blogblogpostagg
: to aggregate blog post, this should be manually added, the module pyquickhelper is preprocessing the documentation to produce pages containing such commandscmdref
: to documentation a script the module makes available on the command linecmdreflist
: to list all commandsepkg
: avoid repeating the same references in many placesexref
: to add an exampleexreflist
: to list all examplefaqref
: to add a FAQfaqreflist
: to list all FAQmathdef
: to add a mathematical definition (or any kind of definition)mathdeflist
: to list all definitionsnbref
: to add a magic commandnbreflist
: to list all magic commandsrunpython
: to run a script and display the output, it can be used to generate documentationsharenet
: to add buttons to share the page on a socal networktodoext
: to add an issue or a work itemtodoextlist
: to list all issues or work item
These commands are documented in Sphinx Extensions.
Notebooks¶
Notebooks in folder _doc/notebooks will be automatically converted into html, rst, pdf, slides formats. That requires latex and pandoc.
Continuous Integration¶
The module is tested with Travis, AppVeyor and local testing with Jenkins for a exhaustive list of unit tests, the documentation, the setup. Everything is fully tested on Windows with the standard distribution and Anaconda. There are three builds definition:
Travis: .travis.yml
AppVeyor: appveyor.yml
Jenkins: .local.jenkins.win.yml
The third file by processed by pyquickhelper itself to produce a series of Jenkins jobs
uploaded to a server. See setup_jenkins_server_yml
to configurate a local Jenkins server.
When modules depend on others modules also being tested, the unit tests and the documentation generation uses a local pypi server (port=8079).
Console Output¶
The plugin Collapsing Sections Plugins can help parsing the output. The following section are added:
---- JENKINS BEGIN UNIT TESTS ----
, *---- JENKINS END UNIT TESTS ----
---- JENKINS BEGIN DOCUMENTATION ----
,---- JENKINS END DOCUMENTATION ----
---- JENKINS BEGIN DOCUMENTATION NOTEBOOKS ----
,---- JENKINS END DOCUMENTATION NOTEBOOKS ----
---- JENKINS BEGIN DOCUMENTATION BLOG ----
,---- JENKINS END DOCUMENTATION BLOG ----
---- JENKINS BEGIN DOCUMENTATION COPY FILES ----
,---- JENKINS END DOCUMENTATION COPY FILES ----
---- JENKINS BEGIN DOCUMENTATION ENCODING ----
,---- JENKINS END DOCUMENTATION ENCODING ----
---- JENKINS BEGIN DOCUMENTATION SPHINX ----
,---- JENKINS END DOCUMENTATION SPHINX ----
---- JENKINS BEGIN WRITE VERSION ----
,---- JENKINS END WRITE VERSION ----
---- JENKINS BEGIN SETUPHOOK ----
,---- JENKINS END SETUPHOOK ----