Sphinx Extensions

I use this module to automate most of the process which compiles and publishes the material for my teachings. One part of that is a series of sphinx extensions. A couple assume that the module they are documenting follows the same design as this one, the others are design free. The whole list is available at List of Sphinx commands added by pyquickhelper.

Simple extensions

Sphinx implements many markups. This module adds a couple of them. Many cheat sheets (see cheat sheet 1, cheat sheet 2, Sphinx Memo) can be found on internet. Most if the time, this extension need a change in the configuration file conf.py before using them to document.

autosignature: display the signature of a class or function

Location: docassert setup.

In conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinx_autosignature']

Sometimes you need to show the signature of a function twice in your documentation. However, the instruction .. autofunction:: can be added only otherwise it produces two entries in the index with the same id. Assuming, you have used .. autofunction:: somewhere, you can recall the signture of a function or a class by using .. autosignature::. It will automatically add a link to the text added by .. autofunction:: or .. autoclass::.

pyquickhelper.sphinxext.sphinx_autosignature.AutoSignatureDirective (self, name, arguments, options, content, lineno, content_offset, block_text, state, state_machine)

This directive displays a shorter signature than sphinx.ext.autodoc. Available options:

  • nosummary: do not display a summary (shorten)

  • annotation: shows annotation

  • nolink: if False, add a link to a full documentation (produced by sphinx.ext.autodoc)

  • members: shows members of a class

  • path: three options, full displays the full path including submodules, name displays the last name, import displays the shortest syntax to import it (default).

  • debug: diplays debug information

  • syspath: additional paths to add to sys.path before importing, ‘;’ separated list

The signature is not always available for builtin functions or C++ functions depending on the way to bind them to Python. See Set the __text_signature__ attribute of callables.

The signature may not be infered by module inspect if the function is a compiled C function. In that case, the signature must be added to the documentation. It will parsed by autosignature with by function enumerate_extract_signature with regular expressions.

bigger: bigger size

In conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinx_bigger_extension']

This extension just changes the size of a text if the output is HTML.

  • default size

  • size 1

  • size 5

  • size 10

collapse: hide or show a block

Location: collapse setup.

This extension adds a button to hide or show a limited part of the documentation.

In conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinx_collapse_extension']

Show or hide a part of the documentation.

docassert: check list of documented parameters

Location: docassert setup.

This extension does nothing but generating warnings if a function or a class documents a misspelled parameter (not in the signature) or if one parameter is missing from the documentation.

In conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinx_docassert_extension']

Sphinx outputs some warnings:

WARNING: [docassert] '_init' has undocumented parameters 'translator_class' (in 'pyquickhelper\_doc\sphinxdoc\source\pyquickhelper\helpgen\sphinxm_convert_doc_sphinx_helper.py').

epkg: cache references

Location: epkg_role.

In conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinx_epkg_extension']

epkg_dictionary = {
    'pandoc': 'http://johnmacfarlane.net/pandoc/',                                       # 1
    'pandas': ('http://pandas.pydata.org/pandas-docs/stable/',                           # 2
        ('http://pandas.pydata.org/pandas-docs/stable/generated/pandas.{0}.html', 1)),   # 3
    }

The variable epkg_dictionary stores the list of url to display. It can be a simple string or a list of possibililies with multiple parameters. The three options above can used like this. The last one allows one parameter separated by :.

The last link is broken before the current file is not python file but a rst. The file extension must be specified. For some websites, url and functions do not follow the same rule. A function must be used in this case to handle the mapping.

def weird_mapping(input):
    # The function receives whatever is between `...`.
    ...
    return anchor, url

This function must be placed at the end or be the only available option.

epkg_dictionary = { 'weird_site': weird_mapping }

However, because it is impossible to use a function as a value in the configuration because pickle does not handle this scenario (see PicklingError on environment when config option value is a callable), my_custom_links needs to be replaced by: ("module_where_it_is_defined.my_custom_links", None). The role epkg will import it based on its name.

postcontents: dynamic contents

Location: PostContentsDirective.

In conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinx_postcontents_extension']

The directive .. contents:: display a short table of contents with what Sphinx knows when entering the page. It will not include any title an instruction could dynamically add to the page. Typically:

.. runpython::
    :rst:

    print("Dynamic title")
    print("+++++++++++++")

This title added by the instruction runpython: execute a script is not considered by .. contents::. The main reason is the direction resolves titles when entering the page and not after the doctree was modified. The directive .. postcontents:: inserts a placeholder in the doctree. It is filled by function transform_postcontents before the final page is created (event 'doctree-resolved'). It looks into the page and adds a link to each local sections.

runpython: execute a script

Location: RunPythonDirective.

In conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinxext_runpython_extension']

Documentation means many examples which needs to be updated when a change happen unless the documentation runs the example itself and update its output. That’s what this directive does. It adds as raw text whatever comes out throught the standard output.

<<<

import os
for i, name in enumerate(os.listdir(".")):
    print(i, name)

>>>

    0 source
    1 build
    2 _notebook_dumps

The output can also be compiled as RST format and the code can be hidden. It is useful if the documentation is a copy/paste of some external process or function. This function can be directly called from the documentation. The output must be converted into RST format. It is then added to the documentation. It is quite useful to display the version of some installed modules.

  • file 0: source

  • file 1: build

  • file 2: _notebook_dumps

If the code throws an exception (except a syntax error), it can be caught by adding the option :exception:. The directive displays the traceback.

<<<

import os
for i, name in enumerate(os.listdir("not existing")):
    pass

>>>

    
    [runpythonerror]
    
    Traceback (most recent call last):
        exec(obj, globs, loc)
      File "", line 6, in <module>
      File "", line 4, in run_python_script_139983484439720
    FileNotFoundError: [Errno 2] No such file or directory: 'not existing'

The directive can also be used to display images with a tweak however. It consists in writing rst code. The variable __WD__ indicates the local directory.

The image needs to be save in the same folder than the rst file.

tutorial/oo.pngn:width:201px

Option :toggle: can hide the code or the output or both but let the user unhide it by clicking on a button.

<<<

for i in range(0, 10):
    print("i=", i)

The last option of runpython allows the user to keep some context from one execution to the next one.

<<<

a_to_keep = 5
print("a_to_keep", "=", a_to_keep)

>>>

    a_to_keep = 5

<<<

a_to_keep += 5
print("a_to_keep", "=", a_to_keep)

>>>

    a_to_keep = 10

sphinx-autorun offers a similar service except it cannot produce compile RST content, hide the source and a couple of other options.

tpl_role: template extension

Location: tpl_role.

In conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinxext_template_extension']

This extension is useful whenever there is a recurrent text or a recurrent pattern in the documentation. Typically, a link which depends on a parameter,

:tpl:`template_name,p1=v2, p2=v2, ...`

The template must be defined in the configuration file:

tpl_template = {'template_name': 'some template'}

template_name can be a template (mako or jinja2) or even a function:

tpl_template = {'py':python_link_doc}

The link ftplib.FTP.storbinary was generated by the snippet on the sidebar based on function python_link_doc.

Bloc extensions

They pretty much follows the same design. They highlight a paragraph and this paragraph can be recalled anywhere on another page. Some options differs depending on the content.

Example: faqref

Location: FaqRef.

In conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinx_faqref_extension']

faqref_include_faqrefs = True

This extension adds a todo:

How to add a FAQ?

Description of the issue.

The tag is important when recalling all of these. You can also an internal reference to it with option :lid:. Option :contents: add a list of all nodes @see cl faqref_node included in the list.

  1. How to add a FAQ?

How to add a FAQ?

Description of the issue.

(original entry : sphinx.rst, line 518)

List of bloc extensions

  • blocref: to add a definition (or any kind of definition)

  • cmdref: to documentation a script the module makes available on the command line

  • exref: to add an example

  • faqref: to add a FAQ

  • mathdef: to add a mathematical definition (or any kind of definition)

  • nbref: to add a magic command

  • todoext: to add an issue or a work item

If same design as pyquickhelper

pyquickhelper was created to automate the creation of the documentation for a python module. It does what this extension sphinx-automodapi does and a little bit more:

  • It automatically converts notebooks into RST, HTML, and slides. The RST format is included in the documentation and links to the other format are added.

  • It automatically creates a notebook gallery and an example gallery.

  • It creates a RST pages for each source file in subfoldeer src.

  • It converts javadoc style into Sphinx style.

  • It handles a blog.

This design is described by an empty module:

Blog Post

I added this extension to write some news connected to the module but probably not true anymore in a couple of years. Blog post can added as a file following the template _doc/sphinxdoc/source/blog/<year>/YYYY-MM-DD_anything.rst.

.. blogpost::
    :title: The title of the post
    :keywords: documentation, startup
    :date: 2017-05-21
    :categories: documentation
    :lid: id-for-reference

    Content of the post.

Parameters

Finally, I tried different styles to document a function. Most of them produce the same output. That’s the purpose of the module: module helpgen._fake_function_to_documentation.

Different styles:

f1:

def f1(a, b):
   """
    Addition 1

    @param      a       parameter a
    @param      b       parameter b
    @return             ``a+b``
    """
    return a + b

f2:

def f2(a, b):
    """Addition 2
    @param      a       parameter a
    @param      b       parameter b
    @return             ``a+b``"""
    return a + b

f3:

def f3(a, b):
    """
    Addition 3

    :param a: parameter a
    :param b: parameter a
    :returns: ``a+b``
    """
    return a + b

f4:

def f4(a, b):
    """Addition 4
    :param a: parameter a
    :param b: parameter a
    :returns: ``a+b``"""
    return a + b

f5:

def f5(a, b):
    """
    Addition 5

    Parameters
    ----------

    a: parameter a

    b: parameter b

    Returns
    -------
    ``a+b``
    """
    return a + b

f6:

def f6(a, b):
    """
    Addition 6

    Args:
        a: parameter a
        b: parameter b

    Returns:
        ``a+b``
    """

For developpers: unit test an extension

I did not find any easy solution to test a Sphinx extension I create. The main idea consists in mocking Sphinx. It works to some extend. Sphinx is also quite difficult to run in memory. Every thing is design to use files. I finally decided to spend some time on Sphinx to be able to run it to convert a RST into HTML and RST. That’s the purpose of the next function:

pyquickhelper.helpgen.rst2html (s, fLOG = <function noLOG at 0x7f508d947620>, writer = ‘html’, keep_warnings = False, directives = None, language = ‘en’, layout = ‘docutils’, document_name = ‘<<string>>’, external_docnames = None, filter_nodes = None, new_extensions = None, update_builder = None, ret_doctree = False, load_bokeh = False, destination = None, destination_path = None, options)

Converts a string from RST into HTML format or transformed RST.

The HTML conversion is quite difficult to read:

<<<

from textwrap import dedent
from pyquickhelper.helpgen import rst2html

text = """

.. faqref::
    :title: How to add a FAQ?
    :tag: faqexample2

    Some description.

.. faqreflist::
    :tag: faqexample2
    :contents:

"""

text = dedent(text)
conv = rst2html(text)
print(conv)

>>>

    somewhere/workspace/pyquickhelper/pyquickhelper_UT_37_std/_doc/sphinxdoc/source/pyquickhelper/helpgen/sphinxm_convert_doc_sphinx_helper.py:209: RemovedInSphinx40Warning: The order of arguments for HTML5Translator has been changed. Please give "document" as 1st and "builder" as 2nd.
      HTMLTranslator.__init__(self, builder, *args, **kwds)
    <?xml version="1.0" encoding="utf-8" ?>
    <!DOCTYPE html>
    <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
    <head>
    <meta charset="utf-8"/>
    <meta name="generator" content="Docutils 0.15.2: http://docutils.sourceforge.net/" />
    <title>&lt;string&gt;</title>
    <link rel="stylesheet" href="/usr/local/lib/python3.7/site-packages/docutils/writers/html4css1/html4css1.css" type="text/css" />
    </head>
    <body>
    <div class="document">
    <div class="admonition-faqref faqref_node admonition" id="indexfaqref-faqexample20">
    <p class="admonition-title">How to add a FAQ?</p>
    <p>Some description.</p>
    </div>
    <div class="admonition" id="indexfaqreflistlist-0">
    </div>
    </div>
    </body>
    </html>

That’s why I prefer RST:

<<<

from textwrap import dedent
from pyquickhelper.helpgen import rst2html

text = """

.. faqref::
    :title: How to add a FAQ?
    :tag: faqexample2

    Some description.

.. faqreflist::
    :tag: faqexample2
    :contents:

"""

text = dedent(text)
conv = rst2html(text, writer="rst")
print(conv)

>>>

    .. _indexfaqref-faqexample20:
    
    How to add a FAQ?: Some description.
    
    .. _indexfaqreflistlist-0:

The function does not seem to show anything for the instruction .. faqreflist:: because it is only calling docutils without using everything Sphinx adds to it. Let’s change that.

<<<

from textwrap import dedent
from pyquickhelper.helpgen import rst2html

text = """

.. faqref::
    :title: How to add a FAQ?
    :tag: faqexample2

    Some description.

.. faqreflist::
    :tag: faqexample2
    :contents:

"""

text = dedent(text)
conv = rst2html(text, writer="rst", layout="sphinx")
print(conv)

>>>

    .. _indexfaqref-faqexample20:
    
    How to add a FAQ?: Some description.
    
    .. _indexfaqreflistlist-0:
    
    1. `How to add a FAQ? <#index-faqref-0-0>`_
    
    How to add a FAQ?: Some description.
    
    (`original entry <#indexfaqref-faqexample20>`_ : <string>, line 3)

You can see now what the directive produces once the tree of nodes (doctree) is unfold. It is easy to write a unit test based on that. The first part is the rst2html, the second part is a ReST builder in extension rst_builder. To use it, just add it to the list of extensions in conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinx_rst_builder']

gitlog: to see the last modification

Location: gitlog_role.

In conf.py:

extensions = [ ...
    'pyquickhelper.sphinxext.sphinx_gitlog_extension']

It adds the date of last modification of the current based on the last commit (if git is used).

"Sun Jun 16 11:33:44 2019 +0200"