.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/plot_parallel_process_concurrent.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_plot_parallel_process_concurrent.py: .. _l-example-dot-process: Parallelization of a dot product with processes (concurrent.futures) ==================================================================== Uses processes to parallelize a dot product is not a very solution because processes do not share memory, they need to exchange data. This parallelisation is efficient if the ratio *exchanged data / computation time* is low. This example uses :epkg:`concurrent.futures`. The cost of creating new processes is also significant. .. GENERATED FROM PYTHON SOURCE LINES 15-42 .. code-block:: default import numpy from tqdm import tqdm from pandas import DataFrame import matplotlib.pyplot as plt import concurrent.futures as cf from td3a_cpp.tools import measure_time def parallel_numpy_dot(va, vb, max_workers=2): if max_workers == 2: with cf.ThreadPoolExecutor(max_workers=max_workers) as e: m = va.shape[0] // 2 f1 = e.submit(numpy.dot, va[:m], vb[:m]) f2 = e.submit(numpy.dot, va[m:], vb[m:]) return f1.result() + f2.result() elif max_workers == 3: with cf.ThreadPoolExecutor(max_workers=max_workers) as e: m = va.shape[0] // 3 m2 = va.shape[0] * 2 // 3 f1 = e.submit(numpy.dot, va[:m], vb[:m]) f2 = e.submit(numpy.dot, va[m:m2], vb[m:m2]) f3 = e.submit(numpy.dot, va[m2:], vb[m2:]) return f1.result() + f2.result() + f3.result() else: raise NotImplementedError() .. GENERATED FROM PYTHON SOURCE LINES 43-44 We check that it returns the same values. .. GENERATED FROM PYTHON SOURCE LINES 44-51 .. code-block:: default va = numpy.random.randn(100).astype(numpy.float64) vb = numpy.random.randn(100).astype(numpy.float64) print(parallel_numpy_dot(va, vb), numpy.dot(va, vb)) .. rst-class:: sphx-glr-script-out .. code-block:: none -7.307949549275689 -7.307949549275692 .. GENERATED FROM PYTHON SOURCE LINES 52-53 Let's benchmark. .. GENERATED FROM PYTHON SOURCE LINES 53-70 .. code-block:: default res = [] for n in tqdm([100000, 1000000, 10000000, 100000000]): va = numpy.random.randn(n).astype(numpy.float64) vb = numpy.random.randn(n).astype(numpy.float64) m1 = measure_time('dot(va, vb, 2)', dict(va=va, vb=vb, dot=parallel_numpy_dot)) m2 = measure_time('dot(va, vb)', dict(va=va, vb=vb, dot=numpy.dot)) res.append({'N': n, 'numpy.dot': m2['average'], 'futures': m1['average']}) df = DataFrame(res).set_index('N') print(df) df.plot(logy=True, logx=True) plt.title("Parallel / numpy dot") .. image-sg:: /auto_examples/images/sphx_glr_plot_parallel_process_concurrent_001.png :alt: Parallel / numpy dot :srcset: /auto_examples/images/sphx_glr_plot_parallel_process_concurrent_001.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0/4 [00:00` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_parallel_process_concurrent.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_