module fctmr.pyparallel_fctmr#

Short summary#

module sparkouille.fctmr.pyparallel_fctmr

joblib uses a module not documented in the official Python documentation: Python’s undocumented ThreadPool.

source on GitHub

Functions#

function

truncated documentation

pyparallel_mapper

Applies function fct to a generator. Relies on ThreadPool.

Documentation#

joblib uses a module not documented in the official Python documentation: Python’s undocumented ThreadPool.

source on GitHub

sparkouille.fctmr.pyparallel_fctmr.pyparallel_mapper(fct, gen, threads=None)#

Applies function fct to a generator. Relies on ThreadPool.

Paramètres:

  • fct – function

  • gen – generator

  • threads – number of threads

Renvoie:

generator

If the number of threads is None, it is replaced by os.cpu_count() or 1 (see multiprocessing.pool).

mapper

<<<

from sparkouille.fctmr.pyparallel_fctmr import pyparallel_mapper
res = pyparallel_mapper(lambda x: x + 1, [4, 5])
print(list(res))

>>>

    /usr/local/lib/python3.9/multiprocessing/pool.py:265: ResourceWarning: unclosed running multiprocessing pool <multiprocessing.pool.ThreadPool state=RUN pool_size=8>
      _warn(f"unclosed running multiprocessing pool {self!r}",
    ResourceWarning: Enable tracemalloc to get the object allocation traceback
    [5, 6]

Unfortunately, the parallelization is not following the map/reduce concept in a sense that the function generates an intermediate list and creates an iterator on it.

source on GitHub