When to parallelize?#

That is the question. Parallize computation takes some time to set up, it is not the right solution in every case. The following example studies the parallelism introduced into the runtime of TreeEnsembleRegressor to see when it is best to do it.

from pprint import pprint
import numpy
from pandas import DataFrame
import matplotlib.pyplot as plt
from tqdm import tqdm
from sklearn import config_context
from sklearn.datasets import make_regression
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import train_test_split
from cpyquickhelper.numbers import measure_time
from pyquickhelper.pycode.profiling import profile
from mlprodict.onnx_conv import to_onnx, register_rewritten_operators
from mlprodict.onnxrt import OnnxInference
from mlprodict.tools.model_info import analyze_model

Available optimisations on this machine.

from mlprodict.testing.experimental_c_impl.experimental_c import code_optimisation
print(code_optimisation())

Out:

AVX-omp=8

Training and converting a model#

data = make_regression(50000, 20)
X, y = data
X_train, X_test, y_train, y_test = train_test_split(X, y)

hgb = HistGradientBoostingRegressor(max_iter=100, max_depth=6)
hgb.fit(X_train, y_train)
print(hgb)

Out:

HistGradientBoostingRegressor(max_depth=6)

Let’s get more statistics about the model itself.

pprint(analyze_model(hgb))

Out:

{'_predictors.max|tree_.max_depth': 6,
 '_predictors.size': 100,
 '_predictors.sum|tree_.leave_count': 3100,
 '_predictors.sum|tree_.node_count': 6100,
 'train_score_.shape': 101,
 'validation_score_.shape': 101}

And let’s convert it.

register_rewritten_operators()
onx = to_onnx(hgb, X_train[:1].astype(numpy.float32))
oinf = OnnxInference(onx, runtime='python_compiled')
print(oinf)

Out:

OnnxInference(...)
    def compiled_run(dict_inputs, yield_ops=None, context=None):
        if yield_ops is not None:
            raise NotImplementedError('yields_ops should be None.')
        # inputs
        X = dict_inputs['X']
        (variable, ) = n0_treeensembleregressor_1(X)
        return {
            'variable': variable,
        }

The runtime of the forest is in the following object.

print(oinf.sequence_[0].ops_)
print(oinf.sequence_[0].ops_.rt_)

Out:

TreeEnsembleRegressor_1(
    op_type=TreeEnsembleRegressor
    aggregate_function=b'SUM',
    base_values=[1.1238728],
    base_values_as_tensor=[],
    domain=ai.onnx.ml,
    inplaces={},
    ir_version=8,
    n_targets=1,
    nodes_falsenodeids=[30 15  8 ...  0  0  0],
    nodes_featureids=[17 15 14 ...  0  0  0],
    nodes_hitrates=[1. 1. 1. ... 1. 1. 1.],
    nodes_missing_value_tracks_true=[0 1 0 ... 0 0 0],
    nodes_modes=[b'BRANCH_LEQ' b'BRANCH_LEQ' b'BRANCH_LEQ' ... b'LEAF' b'LEAF' b'LEAF'],
    nodes_nodeids=[ 0  1  2 ... 58 59 60],
    nodes_treeids=[ 0  0  0 ... 99 99 99],
    nodes_truenodeids=[1 2 3 ... 0 0 0],
    nodes_values=[-0.15475343  0.0349254  -0.25095537 ...  0.          0.
  0.        ],
    post_transform=b'NONE',
    runtime=None,
    target_ids=[0 0 0 ... 0 0 0],
    target_nodeids=[ 5  6  7 ... 58 59 60],
    target_opset=1,
    target_treeids=[ 0  0  0 ... 99 99 99],
    target_weights=[-39.544518   -24.395248   -17.262438   ...   0.05015844   1.5154086
   2.6214952 ],
)
<mlprodict.onnxrt.ops_cpu.op_tree_ensemble_regressor_p_.RuntimeTreeEnsembleRegressorPFloat object at 0x7f82889edc70>

And the threshold used to start parallelizing based on the number of observations.

print(oinf.sequence_[0].ops_.rt_.omp_N_)

Out:

20

Profiling#

This step involves pyinstrument to measure where the time is spent. Both scikit-learn and mlprodict runtime are called so that the prediction times can be compared.

X32 = X_test.astype(numpy.float32)


def runlocal():
    with config_context(assume_finite=True):
        for i in range(0, 100):
            oinf.run({'X': X32[:1000]})
            hgb.predict(X_test[:1000])


print("profiling...")
txt = profile(runlocal, pyinst_format='text')
print(txt[1])

Out:

profiling...

  _     ._   __/__   _ _  _  _ _/_   Recorded: 03:15:39 AM Samples:  1264
 /_//_/// /_\ / //_// / //_'/ //     Duration: 1.702     CPU time: 13.539
/   _/                      v4.1.1

Program: somewhere/workspace/mlprodict/mlprodict_UT_39_std/_doc/examples/plot_parallelism.py

1.702 profile  ../pycode/profiling.py:457
`- 1.701 runlocal  plot_parallelism.py:91
      [44 frames hidden]  plot_parallelism, sklearn, <built-in>...
         1.017 _predict_from_raw_data  <built-in>:0
         0.396 _run  mlprodict/onnxrt/ops_cpu/op_tree_ensemble_regressor.py:95

Now let’s measure the performance the average computation time per observations for 2 to 100 observations. The runtime implemented in mlprodict parallizes the computation after a given number of observations.

obs = []
for N in tqdm(list(range(2, 21))):
    m = measure_time("oinf.run({'X': x})",
                     {'oinf': oinf, 'x': X32[:N]},
                     div_by_number=True,
                     number=20)
    m['N'] = N
    m['RT'] = 'ONNX'
    obs.append(m)

    with config_context(assume_finite=True):
        m = measure_time("hgb.predict(x)",
                         {'hgb': hgb, 'x': X32[:N]},
                         div_by_number=True,
                         number=15)
    m['N'] = N
    m['RT'] = 'SKL'
    obs.append(m)

df = DataFrame(obs)
num = ['min_exec', 'average', 'max_exec']
for c in num:
    df[c] /= df['N']
df.head()

Out:

  0%|          | 0/19 [00:00<?, ?it/s]
  5%|5         | 1/19 [00:00<00:17,  1.02it/s]
 11%|#         | 2/19 [00:01<00:16,  1.02it/s]
 16%|#5        | 3/19 [00:02<00:15,  1.02it/s]
 21%|##1       | 4/19 [00:03<00:14,  1.02it/s]
 26%|##6       | 5/19 [00:04<00:13,  1.02it/s]
 32%|###1      | 6/19 [00:05<00:12,  1.02it/s]
 37%|###6      | 7/19 [00:06<00:11,  1.02it/s]
 42%|####2     | 8/19 [00:07<00:10,  1.01it/s]
 47%|####7     | 9/19 [00:08<00:09,  1.01it/s]
 53%|#####2    | 10/19 [00:09<00:08,  1.01it/s]
 58%|#####7    | 11/19 [00:10<00:07,  1.01it/s]
 63%|######3   | 12/19 [00:11<00:06,  1.00it/s]
 68%|######8   | 13/19 [00:12<00:05,  1.00it/s]
 74%|#######3  | 14/19 [00:13<00:05,  1.00s/it]
 79%|#######8  | 15/19 [00:14<00:04,  1.00s/it]
 84%|########4 | 16/19 [00:15<00:03,  1.01s/it]
 89%|########9 | 17/19 [00:16<00:02,  1.01s/it]
 95%|#########4| 18/19 [00:17<00:01,  1.01s/it]
100%|##########| 19/19 [00:18<00:00,  1.02s/it]
100%|##########| 19/19 [00:18<00:00,  1.00it/s]
average deviation min_exec max_exec repeat number ttime context_size N RT
0 0.000022 0.000002 0.000021 0.000024 10 20 0.000440 232 2 ONNX
1 0.003227 0.000069 0.003197 0.003290 10 15 0.064546 232 2 SKL
2 0.000018 0.000002 0.000017 0.000019 10 20 0.000531 232 3 ONNX
3 0.002134 0.000020 0.002129 0.002147 10 15 0.064022 232 3 SKL
4 0.000016 0.000003 0.000015 0.000018 10 20 0.000633 232 4 ONNX


Graph.

fig, ax = plt.subplots(1, 2, figsize=(10, 4))
df[df.RT == 'ONNX'].set_index('N')[num].plot(ax=ax[0])
ax[0].set_title("Average ONNX prediction time per observation in a batch.")
df[df.RT == 'SKL'].set_index('N')[num].plot(ax=ax[1])
ax[1].set_title(
    "Average scikit-learn prediction time\nper observation in a batch.")
Average ONNX prediction time per observation in a batch., Average scikit-learn prediction time per observation in a batch.

Out:

Text(0.5, 1.0, 'Average scikit-learn prediction time\nper observation in a batch.')

Gain from parallelization#

There is a clear gap between after and before 10 observations when it is parallelized. Does this threshold depends on the number of trees in the model? For that we compute for each model the average prediction time up to 10 and from 10 to 20.

def parallized_gain(df):
    df = df[df.RT == 'ONNX']
    df10 = df[df.N <= 10]
    t10 = sum(df10['average']) / df10.shape[0]
    df10p = df[df.N > 10]
    t10p = sum(df10p['average']) / df10p.shape[0]
    return t10 / t10p


print('gain', parallized_gain(df))

Out:

gain 1.3077563856143504

Measures based on the number of trees#

We trained many models with different number of trees to see how the parallelization gain is moving. One models is trained for every distinct number of trees and then the prediction time is measured for different number of observations.

tries_set = [2, 5, 8] + list(range(10, 50, 5)) + list(range(50, 101, 10))
tries = [(nb, N) for N in range(2, 21, 2) for nb in tries_set]

training

models = {100: (hgb, oinf)}
for nb in tqdm(set(_[0] for _ in tries)):
    if nb not in models:
        hgb = HistGradientBoostingRegressor(max_iter=nb, max_depth=6)
        hgb.fit(X_train, y_train)
        onx = to_onnx(hgb, X_train[:1].astype(numpy.float32))
        oinf = OnnxInference(onx, runtime='python_compiled')
        models[nb] = (hgb, oinf)

Out:

  0%|          | 0/17 [00:00<?, ?it/s]
  6%|5         | 1/17 [00:00<00:04,  3.88it/s]
 12%|#1        | 2/17 [00:01<00:10,  1.44it/s]
 24%|##3       | 4/17 [00:01<00:04,  2.76it/s]
 29%|##9       | 5/17 [00:03<00:09,  1.25it/s]
 35%|###5      | 6/17 [00:03<00:07,  1.48it/s]
 41%|####1     | 7/17 [00:04<00:08,  1.23it/s]
 47%|####7     | 8/17 [00:05<00:06,  1.43it/s]
 53%|#####2    | 9/17 [00:06<00:06,  1.17it/s]
 59%|#####8    | 10/17 [00:07<00:05,  1.31it/s]
 65%|######4   | 11/17 [00:09<00:06,  1.13s/it]
 71%|#######   | 12/17 [00:10<00:05,  1.19s/it]
 76%|#######6  | 13/17 [00:11<00:04,  1.03s/it]
 82%|########2 | 14/17 [00:11<00:02,  1.05it/s]
 88%|########8 | 15/17 [00:14<00:02,  1.34s/it]
 94%|#########4| 16/17 [00:15<00:01,  1.40s/it]
100%|##########| 17/17 [00:16<00:00,  1.25s/it]
100%|##########| 17/17 [00:16<00:00,  1.03it/s]

prediction time

obs = []

for nb, N in tqdm(tries):
    hgb, oinf = models[nb]
    m = measure_time("oinf.run({'X': x})",
                     {'oinf': oinf, 'x': X32[:N]},
                     div_by_number=True,
                     number=50)
    m['N'] = N
    m['nb'] = nb
    m['RT'] = 'ONNX'
    obs.append(m)

df = DataFrame(obs)
num = ['min_exec', 'average', 'max_exec']
for c in num:
    df[c] /= df['N']
df.head()

Out:

  0%|          | 0/170 [00:00<?, ?it/s]
  5%|4         | 8/170 [00:00<00:02, 72.80it/s]
  9%|9         | 16/170 [00:00<00:02, 61.63it/s]
 14%|#3        | 23/170 [00:00<00:02, 62.99it/s]
 18%|#7        | 30/170 [00:00<00:02, 57.78it/s]
 21%|##1       | 36/170 [00:00<00:02, 52.02it/s]
 25%|##4       | 42/170 [00:00<00:02, 53.95it/s]
 28%|##8       | 48/170 [00:00<00:02, 48.18it/s]
 31%|###1      | 53/170 [00:01<00:02, 44.11it/s]
 35%|###4      | 59/170 [00:01<00:02, 46.89it/s]
 38%|###7      | 64/170 [00:01<00:02, 42.94it/s]
 41%|####      | 69/170 [00:01<00:02, 36.60it/s]
 44%|####4     | 75/170 [00:01<00:02, 41.16it/s]
 47%|####7     | 80/170 [00:01<00:02, 38.69it/s]
 50%|#####     | 85/170 [00:01<00:02, 30.82it/s]
 54%|#####3    | 91/170 [00:02<00:02, 36.41it/s]
 56%|#####6    | 96/170 [00:02<00:02, 35.33it/s]
 59%|#####8    | 100/170 [00:02<00:02, 30.43it/s]
 61%|######1   | 104/170 [00:02<00:02, 28.78it/s]
 64%|######4   | 109/170 [00:02<00:01, 32.80it/s]
 66%|######6   | 113/170 [00:02<00:01, 31.49it/s]
 69%|######8   | 117/170 [00:03<00:01, 26.62it/s]
 71%|#######   | 120/170 [00:03<00:02, 24.14it/s]
 74%|#######4  | 126/170 [00:03<00:01, 29.93it/s]
 76%|#######6  | 130/170 [00:03<00:01, 28.82it/s]
 79%|#######8  | 134/170 [00:03<00:01, 24.22it/s]
 81%|########  | 137/170 [00:03<00:01, 21.88it/s]
 84%|########3 | 142/170 [00:04<00:01, 27.24it/s]
 86%|########5 | 146/170 [00:04<00:00, 27.18it/s]
 88%|########7 | 149/170 [00:04<00:00, 24.69it/s]
 89%|########9 | 152/170 [00:04<00:00, 20.19it/s]
 91%|#########1| 155/170 [00:04<00:00, 20.84it/s]
 94%|#########4| 160/170 [00:04<00:00, 25.73it/s]
 96%|#########5| 163/170 [00:04<00:00, 25.21it/s]
 98%|#########7| 166/170 [00:05<00:00, 22.63it/s]
 99%|#########9| 169/170 [00:05<00:00, 18.24it/s]
100%|##########| 170/170 [00:05<00:00, 31.22it/s]
average deviation min_exec max_exec repeat number ttime context_size N nb RT
0 0.000012 3.233795e-07 0.000012 0.000012 10 50 0.000237 232 2 2 ONNX
1 0.000012 2.966389e-07 0.000012 0.000013 10 50 0.000243 232 2 5 ONNX
2 0.000012 2.617136e-07 0.000012 0.000013 10 50 0.000246 232 2 8 ONNX
3 0.000013 4.098758e-07 0.000012 0.000013 10 50 0.000252 232 2 10 ONNX
4 0.000013 2.807648e-07 0.000013 0.000013 10 50 0.000260 232 2 15 ONNX


Let’s compute the gains.

gains = []
for nb in set(df['nb']):
    gain = parallized_gain(df[df.nb == nb])
    gains.append(dict(nb=nb, gain=gain))

dfg = DataFrame(gains)
dfg = dfg.sort_values('nb').reset_index(drop=True).copy()
dfg
nb gain
0 2 3.319433
1 5 3.058542
2 8 2.867054
3 10 2.737095
4 15 2.477233
5 20 2.265952
6 25 2.095773
7 30 1.997449
8 35 1.870391
9 40 1.800700
10 45 1.722864
11 50 1.668391
12 60 1.578462
13 70 1.529147
14 80 1.473345
15 90 1.395878
16 100 1.344745


Graph.

ax = dfg.set_index('nb').plot()
ax.set_title(
    "Parallelization gain depending\non the number of trees\n(max_depth=6).")
Parallelization gain depending on the number of trees (max_depth=6).

Out:

Text(0.5, 1.0, 'Parallelization gain depending\non the number of trees\n(max_depth=6).')

That does not answer the question we are looking for as we would like to know the best threshold th which defines the number of observations for which we should parallelized. This number depends on the number of trees. A gain > 1 means the parallization should happen Here, even two observations is ok. Let’s check with lighter trees (max_depth=2), maybe in that case, the conclusion is different.

models = {100: (hgb, oinf)}
for nb in tqdm(set(_[0] for _ in tries)):
    if nb not in models:
        hgb = HistGradientBoostingRegressor(max_iter=nb, max_depth=2)
        hgb.fit(X_train, y_train)
        onx = to_onnx(hgb, X_train[:1].astype(numpy.float32))
        oinf = OnnxInference(onx, runtime='python_compiled')
        models[nb] = (hgb, oinf)

obs = []
for nb, N in tqdm(tries):
    hgb, oinf = models[nb]
    m = measure_time("oinf.run({'X': x})",
                     {'oinf': oinf, 'x': X32[:N]},
                     div_by_number=True,
                     number=50)
    m['N'] = N
    m['nb'] = nb
    m['RT'] = 'ONNX'
    obs.append(m)

df = DataFrame(obs)
num = ['min_exec', 'average', 'max_exec']
for c in num:
    df[c] /= df['N']
df.head()

Out:

  0%|          | 0/17 [00:00<?, ?it/s]
  6%|5         | 1/17 [00:00<00:03,  4.61it/s]
 12%|#1        | 2/17 [00:00<00:04,  3.24it/s]
 24%|##3       | 4/17 [00:00<00:02,  5.32it/s]
 29%|##9       | 5/17 [00:01<00:03,  3.37it/s]
 35%|###5      | 6/17 [00:01<00:03,  3.56it/s]
 41%|####1     | 7/17 [00:02<00:03,  3.16it/s]
 47%|####7     | 8/17 [00:02<00:02,  3.36it/s]
 53%|#####2    | 9/17 [00:02<00:02,  2.99it/s]
 59%|#####8    | 10/17 [00:02<00:02,  3.15it/s]
 65%|######4   | 11/17 [00:03<00:02,  2.50it/s]
 71%|#######   | 12/17 [00:03<00:02,  2.42it/s]
 76%|#######6  | 13/17 [00:04<00:01,  2.63it/s]
 82%|########2 | 14/17 [00:04<00:01,  2.75it/s]
 88%|########8 | 15/17 [00:05<00:00,  2.26it/s]
 94%|#########4| 16/17 [00:05<00:00,  2.19it/s]
100%|##########| 17/17 [00:06<00:00,  2.36it/s]
100%|##########| 17/17 [00:06<00:00,  2.79it/s]

  0%|          | 0/170 [00:00<?, ?it/s]
  5%|4         | 8/170 [00:00<00:02, 77.07it/s]
  9%|9         | 16/170 [00:00<00:02, 71.32it/s]
 14%|#4        | 24/170 [00:00<00:02, 70.15it/s]
 19%|#8        | 32/170 [00:00<00:02, 66.79it/s]
 23%|##2       | 39/170 [00:00<00:02, 64.01it/s]
 27%|##7       | 46/170 [00:00<00:01, 63.40it/s]
 31%|###1      | 53/170 [00:00<00:02, 56.80it/s]
 35%|###5      | 60/170 [00:00<00:01, 59.20it/s]
 39%|###9      | 67/170 [00:01<00:01, 54.65it/s]
 43%|####2     | 73/170 [00:01<00:01, 53.13it/s]
 46%|####6     | 79/170 [00:01<00:01, 53.83it/s]
 50%|#####     | 85/170 [00:01<00:01, 45.05it/s]
 54%|#####4    | 92/170 [00:01<00:01, 50.32it/s]
 58%|#####7    | 98/170 [00:01<00:01, 49.34it/s]
 61%|######1   | 104/170 [00:01<00:01, 42.54it/s]
 65%|######4   | 110/170 [00:02<00:01, 46.41it/s]
 68%|######7   | 115/170 [00:02<00:01, 45.43it/s]
 71%|#######   | 120/170 [00:02<00:01, 37.64it/s]
 75%|#######4  | 127/170 [00:02<00:00, 43.24it/s]
 78%|#######7  | 132/170 [00:02<00:00, 42.56it/s]
 81%|########  | 137/170 [00:02<00:00, 34.95it/s]
 85%|########4 | 144/170 [00:02<00:00, 40.68it/s]
 88%|########7 | 149/170 [00:03<00:00, 40.06it/s]
 91%|######### | 154/170 [00:03<00:00, 32.55it/s]
 94%|#########4| 160/170 [00:03<00:00, 37.98it/s]
 97%|#########7| 165/170 [00:03<00:00, 38.42it/s]
100%|##########| 170/170 [00:03<00:00, 29.63it/s]
100%|##########| 170/170 [00:03<00:00, 44.93it/s]
average deviation min_exec max_exec repeat number ttime context_size N nb RT
0 0.000012 3.640651e-07 0.000012 0.000012 10 50 0.000235 232 2 2 ONNX
1 0.000012 6.046737e-07 0.000012 0.000013 10 50 0.000239 232 2 5 ONNX
2 0.000012 3.939902e-07 0.000012 0.000013 10 50 0.000240 232 2 8 ONNX
3 0.000012 2.011198e-07 0.000012 0.000012 10 50 0.000241 232 2 10 ONNX
4 0.000012 2.231737e-07 0.000012 0.000012 10 50 0.000244 232 2 15 ONNX


Measures.

gains = []
for nb in set(df['nb']):
    gain = parallized_gain(df[df.nb == nb])
    gains.append(dict(nb=nb, gain=gain))

dfg = DataFrame(gains)
dfg = dfg.sort_values('nb').reset_index(drop=True).copy()
dfg
nb gain
0 2 3.419412
1 5 3.281398
2 8 3.164646
3 10 3.078026
4 15 2.924500
5 20 2.793549
6 25 2.687950
7 30 2.570847
8 35 2.493058
9 40 2.404950
10 45 2.317088
11 50 2.249583
12 60 2.087625
13 70 1.968180
14 80 1.862171
15 90 1.791309
16 100 1.349174


Graph.

ax = dfg.set_index('nb').plot()
ax.set_title(
    "Parallelization gain depending\non the number of trees\n(max_depth=3).")
Parallelization gain depending on the number of trees (max_depth=3).

Out:

Text(0.5, 1.0, 'Parallelization gain depending\non the number of trees\n(max_depth=3).')

The conclusion is somewhat the same but it shows that the bigger the number of trees is the bigger the gain is and under the number of cores of the processor.

Moving the theshold#

The last experiment consists in comparing the prediction time with or without parallelization for different number of observation.

hgb = HistGradientBoostingRegressor(max_iter=40, max_depth=6)
hgb.fit(X_train, y_train)
onx = to_onnx(hgb, X_train[:1].astype(numpy.float32))
oinf = OnnxInference(onx, runtime='python_compiled')


obs = []
for N in tqdm(list(range(2, 51))):
    oinf.sequence_[0].ops_.rt_.omp_N_ = 100
    m = measure_time("oinf.run({'X': x})",
                     {'oinf': oinf, 'x': X32[:N]},
                     div_by_number=True,
                     number=20)
    m['N'] = N
    m['RT'] = 'ONNX'
    m['PARALLEL'] = False
    obs.append(m)

    oinf.sequence_[0].ops_.rt_.omp_N_ = 1
    m = measure_time("oinf.run({'X': x})",
                     {'oinf': oinf, 'x': X32[:N]},
                     div_by_number=True,
                     number=50)
    m['N'] = N
    m['RT'] = 'ONNX'
    m['PARALLEL'] = True
    obs.append(m)

df = DataFrame(obs)
num = ['min_exec', 'average', 'max_exec']
for c in num:
    df[c] /= df['N']
df.head()

Out:

  0%|          | 0/49 [00:00<?, ?it/s]
  8%|8         | 4/49 [00:00<00:01, 36.04it/s]
 16%|#6        | 8/49 [00:00<00:01, 31.66it/s]
 24%|##4       | 12/49 [00:00<00:01, 27.68it/s]
 31%|###       | 15/49 [00:00<00:01, 25.18it/s]
 37%|###6      | 18/49 [00:00<00:01, 22.79it/s]
 43%|####2     | 21/49 [00:00<00:01, 20.77it/s]
 49%|####8     | 24/49 [00:01<00:01, 19.02it/s]
 53%|#####3    | 26/49 [00:01<00:01, 17.87it/s]
 57%|#####7    | 28/49 [00:01<00:01, 16.80it/s]
 61%|######1   | 30/49 [00:01<00:01, 15.84it/s]
 65%|######5   | 32/49 [00:01<00:01, 14.96it/s]
 69%|######9   | 34/49 [00:01<00:01, 14.14it/s]
 73%|#######3  | 36/49 [00:01<00:00, 13.42it/s]
 78%|#######7  | 38/49 [00:02<00:00, 12.79it/s]
 82%|########1 | 40/49 [00:02<00:00, 12.23it/s]
 86%|########5 | 42/49 [00:02<00:00, 11.70it/s]
 90%|########9 | 44/49 [00:02<00:00, 11.22it/s]
 94%|#########3| 46/49 [00:02<00:00, 10.80it/s]
 98%|#########7| 48/49 [00:03<00:00, 10.41it/s]
100%|##########| 49/49 [00:03<00:00, 15.14it/s]
average deviation min_exec max_exec repeat number ttime context_size N RT PARALLEL
0 0.000016 1.035920e-06 0.000015 0.000017 10 20 0.000312 232 2 ONNX False
1 0.000018 1.513150e-06 0.000017 0.000019 10 50 0.000355 232 2 ONNX True
2 0.000012 1.231526e-06 0.000011 0.000013 10 20 0.000347 232 3 ONNX False
3 0.000012 1.027312e-06 0.000012 0.000013 10 50 0.000363 232 3 ONNX True
4 0.000010 9.711232e-07 0.000009 0.000010 10 20 0.000381 232 4 ONNX False


Graph.

piv = df[['N', 'PARALLEL', 'average']].pivot('N', 'PARALLEL', 'average')
ax = piv.plot(logy=True)
ax.set_title("Prediction time with and without parallelization.")
Prediction time with and without parallelization.

Out:

Text(0.5, 1.0, 'Prediction time with and without parallelization.')

Parallelization is working.

plt.show()

Total running time of the script: ( 1 minutes 1.639 seconds)

Gallery generated by Sphinx-Gallery