module fctmr.pyparallel_fctmr

Short summary

module sparkouille.fctmr.pyparallel_fctmr

:epkg:`joblib` uses a module not documented in the official Python documentation: Python’s undocumented ThreadPool.

source on GitHub

Functions

function

truncated documentation

pyparallel_mapper

Applies function fct to a generator. Relies on ThreadPool.

Documentation

:epkg:`joblib` uses a module not documented in the official Python documentation: Python’s undocumented ThreadPool.

source on GitHub

sparkouille.fctmr.pyparallel_fctmr.pyparallel_mapper(fct, gen, threads=None)[source]

Applies function fct to a generator. Relies on ThreadPool.

Paramètres
  • fct – function

  • gen – generator

  • threads – number of threads

Renvoie

generator

If the number of threads is None, it is replaced by os.cpu_count() or 1 (see multiprocessing.pool).

mapper

<<<

from sparkouille.fctmr.pyparallel_fctmr import pyparallel_mapper
res = pyparallel_mapper(lambda x: x + 1, [4, 5])
print(list(res))

>>>

    [5, 6]

Unfortunately, the parallelization is not following the map/reduce concept in a sense that the function generates an intermediate list and creates an iterator on it.

source on GitHub