.. _l-modules:

Modules
=======

.. contents::
    :local:
    :depth: 1


Summary
+++++++

+--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| module                                           | truncated documentation                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+==================================================+================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================+
| :ref:`__init__ <f-init>`                         | module ``sparkouille``    Around :epkg:`Spark`.      :githublink:`%|py|6`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| :ref:`__init__ <f-initz>`                        | module ``sparkouille.datasets``    Shorcuts to *datasets*      :githublink:`%|py|5`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| :ref:`__init__ <f-initzz>`                       | module ``sparkouille.fctmr``    Shorcuts to *fctmr*      :githublink:`%|py|5`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| :ref:`eurostat <f-eurostat>`                     | module ``sparkouille.datasets.eurostat``    Datasets from :epkg:`Eurostat`.      :githublink:`%|py|6`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| :ref:`fast_parallel_fctmr <f-fastparallelfctmr>` | module ``sparkouille.fctmr.fast_parallel_fctmr``    Simple parallelization of *mapper* and *reducer* based on :epkg:`numba`.  :epkg:`Python` does not easily allow to parallelize functions  as the :epkg:`GIL` blocks most of the tentatives by imposing  a single tunnel for all allocations, creation of :epkg:`python`  objects. The language implements it but in practice it is not.  This file is just a tentative to use :epkg:`numba` to parallelize  a mapper but the number of round trip between :epkg:`python`  and compiled :epkg:`C` makes it difficult to write something  generic.      :githublink:`%|py|14` |
+--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| :ref:`pyparallel_fctmr <f-pyparallelfctmr>`      | module ``sparkouille.fctmr.pyparallel_fctmr``    :epkg:`joblib` uses a module not documented  in the official :epkg:`Python` documentation:  `Python's undocumented ThreadPool <http://lucasb.eyer.be/snips/python-thread-pool.html>`_.      :githublink:`%|py|8`                                                                                                                                                                                                                                                                                                                                                              |
+--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| :ref:`simplefctmr <f-simplefctmr>`               | module ``sparkouille.fctmr.simplefctmr``    Simple *mapper* and *reducer* implemented in :epkg:`Python`      :githublink:`%|py|6`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


.. toctree::
    :maxdepth: 1

    sparkouille/__init__
    sparkouille/datasets/__init__
    sparkouille/datasets/eurostat
    sparkouille/fctmr/__init__
    sparkouille/fctmr/fast_parallel_fctmr
    sparkouille/fctmr/pyparallel_fctmr
    sparkouille/fctmr/simplefctmr