module helpgen.utils_sphinx_doc

Inheritance diagram of pyquickhelper.helpgen.utils_sphinx_doc

Short summary

module pyquickhelper.helpgen.utils_sphinx_doc

various helpers to produce a Sphinx documentation

source on GitHub

Classes

class

truncated documentation

useless_class_UnicodeStringIOThreadSafe

avoid conversion problem between str and char, class protected again Thread issue

Functions

function

truncated documentation

_private_migrating_doxygen_doc

_private_process_one_file

Copies one file from the source to the documentation folder. It processes some comments in doxygen format (@ param, …

add_file_rst

Creates a rst file for every source file.

apply_modification_template

See add_file_rst().

copy_source_files

Copies all sources files (input) into a folder (output), apply on each of them a modification.

doc_checking

Example of a doc string.

filecontent_to_rst

Produces a .rst file which contains the file. It adds a title and a label based on the filename (no folder included). …

fix_incomplete_references

Looks into every file .rst or .py for incomplete reference. Example

migrating_doxygen_doc

Migrates the doxygen documentation to rst format.

prepare_file_for_sphinx_help_generation

Prepares all files for Sphinx generation.

process_copy_images

Looks into every file .rst or .py for images (

produces_indexes

Produces a file for each category of object found in the module.

remove_undesired_part_for_documentation

Some files contains blocs inserted between the two lines:

replace_relative_import_fct

Takes a python file and replaces all relative imports it was able to find by an import which can be processed …

validate_file_for_help

accept or reject a file to be copied in the help folder

Methods

method

truncated documentation

__init__

creates a lock

Documentation

various helpers to produce a Sphinx documentation

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc._private_migrating_doxygen_doc(rows, index_first_line, filename, debug=False, silent=False)[source][source]
pyquickhelper.helpgen.utils_sphinx_doc._private_process_one_file(fullname, to, silent, fmod, replace_relative_import, use_sys)[source][source]

Copies one file from the source to the documentation folder. It processes some comments in doxygen format (@ param, @ return). It replaces relatives imports by a regular import.

Parameters
  • fullname – name of the file

  • to – location (folder)

  • silent – no logs if True

  • fmod – modification functions

  • replace_relative_import – replace relative import

  • use_sysremove_undesired_part_for_documentation

Returns

extension, number of lines, number of lines in documentation

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.add_file_rst(rootm, store_obj, actions, template='\n__FULLNAME_UNDERLINED__\n\n\n\n\n.. inheritance-diagram:: __FULLNAMENOEXT__\n\n\nShort summary\n+++++++++++++\n\n__DOCUMENTATION__\n\n\n__CLASSES__\n\n__FUNCTIONS__\n\n__PROPERTIES__\n\n__STATICMETHODS__\n\n__METHODS__\n\nDocumentation\n+++++++++++++\n\n.. automodule:: __FULLNAMENOEXT__\n :members:\n :special-members: __init__\n :show-inheritance:\n\n__ADDEDMEMBERS__\n\n', rootrep=('_doc.sphinxdoc.source.pyquickhelper.', ''), fmod=<function <lambda>>, softfile=<function <lambda>>, mapped_function=None, indexes=None, additional_sys_path=None, fLOG=<function noLOG>)[source][source]

Creates a rst file for every source file.

Parameters
  • rootm – root of the module (for relative import)

  • store_obj – to keep table of all objects

  • actions – output from copy_source_files

  • templaterst template to produce

  • rootrep – file name in the documentation contains some folders which are not desired in the documentation

  • fmod – applies modification to the instanciated template

  • softfile – softfile is a function (f : filename –> True or False), when it is True, the documentation is lighter (no special members)

  • mapped_function – list of 2-tuple (pattern, function). Every file matching the pattern will be copied to the documentation folder, its content will be sent to the function and will produce a file <filename>.rst. Example: [ (".*[.]sql$", filecontent_to_rst) ] The function takes two parameters: full_filename, content_filename. It returns a string (the rst file) or a tuple (rst file, short description). By default (if function is None), the function filecontent_to_rst will be called except for .rst file for which nothing is done.

  • indexes – to index some information { dictionary label:IndexInformation (…) }, the function populates it

  • additional_sys_path – additional path to include to sys.path before importing a module (will be removed afterwards)

  • fLOG – logging function

Returns

list of written files stored in RstFileHelp

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.apply_modification_template(rootm, store_obj, template, fullname, rootrep, softfile, indexes, additional_sys_path, fLOG=<function noLOG>)[source][source]

See add_file_rst.

Parameters
  • rootm – root of the module

  • store_obj – keep track of all objects extracted from the module

  • template – rst template to produce

  • fullname – full name of the file

  • rootrep – file name in the documentation contains some folders which are not desired in the documentation

  • softfile – a function (f : filename –> True or False), when it is True, the documentation is lighter (no special members)

  • indexes – dictionary with the label and some information (IndexInformation)

  • additional_sys_path – additional path to include to sys.path before importing a module (will be removed afterwards)

  • fLOG – logging function

Returns

content of a .rst file

Why doesn’t the documentation show compiled submodules?

The instruction .. automodule:: <name> only shows objects obj which verify obj.__module__ == name. This is always the case for modules written in Python but not necessarily for module compiled from C language. When the module is declared, the following structure contains the module name in second position. This name must not be the submodule shortname but the name the module has is the package. The C file pyquickhelper/helpgen/compiled.c implements submodule pyquickhelper.helpgen.compiled, this value must replace <fullname> in the structure below, not simply compiled.

static struct PyModuleDef moduledef = {
        PyModuleDef_HEAD_INIT,
        "<fullname>",
        "Helper for parallelization with threads with C++.",
        sizeof(struct module_state),
        fonctions,
        NULL,
        threader_module_traverse,
        threader_module_clear,
        NULL
};

Warning

This function still needs some improvments for C++ modules on MacOSX.

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.copy_source_files(input, output, fmod=<function <lambda>>, silent=False, filter=None, remove=True, softfile=<function <lambda>>, fexclude=<function <lambda>>, addfilter=None, replace_relative_import=False, copy_add_ext=None, use_sys=None, fLOG=<function fLOG>)[source][source]

Copies all sources files (input) into a folder (output), apply on each of them a modification.

Parameters
  • input – input folder

  • output – output folder (it will be cleaned each time)

  • fmod – modifies the content of each file, this function takes a string and returns a string

  • silent – if True, do not stop when facing an issue with doxygen documentation

  • filter – if None, process only file related to python code, otherwise, use this filter to select file (regular expression). If this parameter is None or is empty, the default value is something like: "(.+[.]py$)|(.+[.]pyd$)|(.+[.]cpp$)|(.+[.]h$)|(.+[.]dll$))".

  • remove – if True, remove every files in the output folder first

  • softfile – softfile is a function (f : filename –> True or False), when it is True, the documentation is lighter (no special members)

  • fexclude – function to exclude some files from the help

  • addfilter – additional filter, it should look like: "(.+[.]pyx$)|(.+[.]pyh$)"

  • replace_relative_import – replace relative import

  • copy_add_ext – additional extension file to copy

  • use_sys – see remove_undesired_part_for_documentation

  • fLOG – logging function

Returns

list of copied files

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.doc_checking()[source][source]

Example of a doc string.

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.filecontent_to_rst(filename, content)[source][source]

Produces a .rst file which contains the file. It adds a title and a label based on the filename (no folder included).

Parameters
  • filename – filename

  • content – content

Returns

new content

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.fix_incomplete_references(folder_source, store_obj, issues=None, fLOG=<function fLOG>)[source][source]

Looks into every file .rst or .py for incomplete reference. Example:

:class:`name`  --> :class:`name <...>`.
Parameters
  • folder_source – folder where to look for sources

  • store_obj – container for indexed objects

  • issues – if not None (a list), it will add issues (function, message)

  • fLOG – logging function

Returns

list of fixed references

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.migrating_doxygen_doc(content, filename, silent=False, log=False, debug=False)[source][source]

Migrates the doxygen documentation to rst format.

Parameters
  • content – file content

  • filename – filename (to display useful error messages)

  • silent – if silent, do not raise an exception

  • log – if True, write some information in the logs (not only exceptions)

  • debug – display more information on the output if True

Returns

statistics, new content file

Function private_migrating_doxygen_doc enumerates the list of conversion which will be done.

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.prepare_file_for_sphinx_help_generation(store_obj, input, output, subfolders, fmod_copy=<function <lambda>>, template='\n__FULLNAME_UNDERLINED__\n\n\n\n\n.. inheritance-diagram:: __FULLNAMENOEXT__\n\n\nShort summary\n+++++++++++++\n\n__DOCUMENTATION__\n\n\n__CLASSES__\n\n__FUNCTIONS__\n\n__PROPERTIES__\n\n__STATICMETHODS__\n\n__METHODS__\n\nDocumentation\n+++++++++++++\n\n.. automodule:: __FULLNAMENOEXT__\n :members:\n :special-members: __init__\n :show-inheritance:\n\n__ADDEDMEMBERS__\n\n', rootrep=('_doc.sphinxdoc.source.project_name.', ''), fmod_res=<function <lambda>>, silent=False, optional_dirs=None, softfile=<function <lambda>>, fexclude=<function <lambda>>, mapped_function=None, fexclude_index=<function <lambda>>, issues=None, additional_sys_path=None, replace_relative_import=False, module_name=None, copy_add_ext=None, use_sys=None, auto_rst_generation=True, fLOG=<function fLOG>)[source][source]

Prepares all files for Sphinx generation.

Parameters
  • store_obj – to keep track of all objects, it should be a dictionary

  • input – input folder

  • output – output folder (it will be cleaned each time)

  • subfolders – list of subfolders to copy from input to output, two cases: * a string input/<sub> –> output/<sub> * a tuple input/<sub[0]> –> output/<sub[1]>

  • fmod_copy – modifies the content of each file, this function takes a string and the filename and returns a string f(content, filename) --> string

  • template – rst template to produce

  • rootrep – file name in the documentation contains some folders which are not desired in the documentation

  • fmod_res – applies modification to the instanciated template

  • silent – if True, do not stop when facing an issue with doxygen migration

  • optional_dirs – list of tuple with a list of folders (source, copy, filter) to copy for the documentation, example: ( <folder_help>, "coverage", ".*" )

  • softfile – softfile is a function (f : filename –> True or False), when it is True, the documentation is lighter (no special members)

  • fexclude – function to exclude some files from the help

  • fexclude_index – function to exclude some files from the indices

  • mapped_function – list of 2-tuple (pattern, function). Every file matching the pattern will be copied to the documentation folder, its content will be sent to the function and will produce a file <filename>.rst. Example: [ (".*[.]sql$", filecontent_to_rst) ] The function takes two parameters: full_filename, content_filename. It returns a string (the rst file) or a tuple (rst file, short description). By default (if function is None), the function filecontent_to_rst will be called.

  • issues – if not None (a list), the function will store some issues here.

  • additional_sys_path – additional paths to includes to sys.path when import a module (will be removed afterwards)

  • replace_relative_import – replace relative import

  • module_name – module name (cannot be None)

  • copy_add_ext – additional file extension to copy

  • use_sysremove_undesired_part_for_documentation

  • auto_rst_generation – add a file .rst for each source file

  • fLOG – logging function

Returns

list of written files stored in RstFileHelp

Example:

prepare_file_for_sphinx_help_generation (
            {},
            ".",
            "_doc/sphinxdoc/source/",
            subfolders      = [
                                ("src/" + project_var_name,
                                 project_var_name),
                                 ],
            silent          = True,
            rootrep         = ("_doc.sphinxdoc.source.%s." %
                               (project_var_name,), ""),
            optional_dirs   = optional_dirs,
            mapped_function = [ (".*[.]tohelp$", None) ] )

It produces a file with the number of lines and files per extension.

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.process_copy_images(folder_source, folder_images)[source][source]

Looks into every file .rst or .py for images (.. image:: imagename), if this image was found in directory folder_images, then the image is copied closes to the file.

Parameters
  • folder_source – folder where to look for sources

  • folder_images – folder where to look for images

Returns

list of copied images

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.produces_indexes(store_obj, indexes, fexclude_index, titles=None, correspondances=None, fLOG=<function fLOG>)[source][source]

Produces a file for each category of object found in the module.

Parameters
  • store_obj – list of collected object, it is a dictionary key : ModuleMemberDoc or key : [ list of ModuleMemberDoc ]

  • indexes – list of things to index, dictionary { label : IndexInformation }

  • fexclude_index – to exclude files from the indices

  • titles – each type is mapped to a title to add to the rst file

  • correspondances – each type is mapped to a label to add to the rst file

  • fLOG – logging function

Returns

dictionary: { type : rst content of the index }

Default values if titles of correspondances is None:

title = {"method": "Methods",
         "staticmethod": "Static Methods",
         "property": "Properties",
         "function": "Functions",
         "class": "Classes",
         "module": "Modules"}

correspondances = {"method": "l-methods",
                   "function": "l-functions",
                   "staticmethod": "l-staticmethods",
                   "property": "l-properties",
                   "class": "l-classes",
                   "module": "l-modules"}

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.remove_undesired_part_for_documentation(content, filename, use_sys)[source][source]

Some files contains blocs inserted between the two lines:

  • # -- HELP BEGIN EXCLUDE --

  • # -- HELP END EXCLUDE --

Those lines will be commented out.

Parameters
  • content – file content

  • filename – for error message

  • use_sys – string or None, enables, disables a section based on variables added to sys module

Returns

modified file content

If the parameter use_sys is false, the section of code will be commented out. If true, the section can be enabled. It relies on the following code:

import sys
if hasattr(sys, "<use_sys>") and sys.<use_sys>:

    # section to enable or disables

The string <use_sys> will be replaced by the value of parameter use_sys.

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.replace_relative_import_fct(fullname, content=None)[source][source]

Takes a python file and replaces all relative imports it was able to find by an import which can be processed by Python if the file were the main file.

Parameters
  • fullname – name of the file

  • content – a preprocessed content of the file of the content if it is None

Returns

content of the file without relative imports

Changed in version 1.8: Does not change imports in comments.

source on GitHub

class pyquickhelper.helpgen.utils_sphinx_doc.useless_class_UnicodeStringIOThreadSafe[source][source]

Bases: str

avoid conversion problem between str and char, class protected again Thread issue

source on GitHub

creates a lock

source on GitHub

__init__()[source][source]

creates a lock

source on GitHub

pyquickhelper.helpgen.utils_sphinx_doc.validate_file_for_help(filename, fexclude=<function <lambda>>)[source][source]

accept or reject a file to be copied in the help folder

Parameters
  • filename – filename

  • fexclude – function to exclude some files

Returns

boolean

source on GitHub