.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/plot_long_parallel_process_joblib.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_plot_long_parallel_process_joblib.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_plot_long_parallel_process_joblib.py:


.. _l-example-dot-joblib:

Parallelization of a dot product with processes (joblib)
========================================================

Uses processes to parallelize a dot product is not
a very solution because processes do not share memory,
they need to exchange data. This parallelisation
is efficient if the ratio *exchanged data / computation time*
is low. :epkg:`joblib` is used by :epkg:`scikit-learn`.
The cost of creating new processes is also significant.

.. GENERATED FROM PYTHON SOURCE LINES 15-37

.. code-block:: default


    import numpy
    from tqdm import tqdm
    from pandas import DataFrame
    import matplotlib.pyplot as plt
    from joblib import Parallel, delayed
    from td3a_cpp.tools import measure_time


    def parallel_dot_joblib(va, vb, max_workers=2):
        dh = va.shape[0] // max_workers
        k = 2
        dhk = dh // k
        if dh != float(va.shape[0]) / max_workers:
            raise RuntimeError("size must be a multiple of max_workers.")

        r = Parallel(n_jobs=max_workers, backend="loky")(
            delayed(numpy.dot)(va[i * dhk:i * dhk + dhk],
                               vb[i * dhk:i * dhk + dhk])
            for i in range(max_workers * k))
        return sum(r)








.. GENERATED FROM PYTHON SOURCE LINES 38-39

We check that it returns the same values.

.. GENERATED FROM PYTHON SOURCE LINES 39-46

.. code-block:: default



    va = numpy.random.randn(100).astype(numpy.float64)
    vb = numpy.random.randn(100).astype(numpy.float64)
    print(parallel_dot_joblib(va, vb), numpy.dot(va, vb))






.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    -9.082747130461469 -9.082747130461467




.. GENERATED FROM PYTHON SOURCE LINES 47-48

Let's benchmark.

.. GENERATED FROM PYTHON SOURCE LINES 48-68

.. code-block:: default



    res = []
    for n in tqdm([1000, 2000]):
        va = numpy.random.randn(n).astype(numpy.float64)
        vb = numpy.random.randn(n).astype(numpy.float64)

        m1 = measure_time('dot(va, vb, 2)',
                          dict(va=va, vb=vb, dot=parallel_dot_joblib),
                          repeat=1)
        m2 = measure_time('dot(va, vb)',
                          dict(va=va, vb=vb, dot=numpy.dot))
        res.append({'N': n, 'numpy.dot': m2['average'],
                    'joblib': m1['average']})

    df = DataFrame(res).set_index('N')
    print(df)
    df.plot(logy=True, logx=True)
    plt.title("Parallel / numpy dot")




.. image-sg:: /auto_examples/images/sphx_glr_plot_long_parallel_process_joblib_001.png
   :alt: Parallel / numpy dot
   :srcset: /auto_examples/images/sphx_glr_plot_long_parallel_process_joblib_001.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|          | 0/2 [00:00<?, ?it/s]
     50%|#####     | 1/2 [00:00<00:00,  1.46it/s]
    100%|##########| 2/2 [00:01<00:00,  1.45it/s]
    100%|##########| 2/2 [00:01<00:00,  1.45it/s]
          numpy.dot    joblib
    N                        
    1000   0.000010  0.013563
    2000   0.000011  0.013500

    Text(0.5, 1.0, 'Parallel / numpy dot')



.. GENERATED FROM PYTHON SOURCE LINES 69-70

The parallelisation is inefficient.

.. GENERATED FROM PYTHON SOURCE LINES 70-72

.. code-block:: default


    plt.show()








.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  5.070 seconds)


.. _sphx_glr_download_auto_examples_plot_long_parallel_process_joblib.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example




    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_long_parallel_process_joblib.py <plot_long_parallel_process_joblib.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_long_parallel_process_joblib.ipynb <plot_long_parallel_process_joblib.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_