module `cli.validate`#

Short summary#

module mlprodict.cli.validate

Command line about validation of prediction runtime.

Functions#

function	truncated documentation
`_finalize`
`_validate_runtime_dict`
`_validate_runtime_separate_process`
`benchmark_doc`	Runs the benchmark published into the documentation (see Availability of scikit-learn model for runtime onnxruntime1 and Availability of scikit-learn model for runtime python_compiled). …
`latency`	Measures the latency of a model (python API).
`validate_runtime`	Walks through most of scikit-learn operators or model or predictor or transformer, tries to convert them …

Documentation#

Command line about validation of prediction runtime.

source on GitHub

mlprodict.cli.validate._finalize(rows, out_raw, out_summary, verbose, models, out_graph, fLOG)#

mlprodict.cli.validate._validate_runtime_dict(kwargs)#

mlprodict.cli.validate._validate_runtime_separate_process(**kwargs)#

mlprodict.cli.validate.benchmark_doc(runtime, black_list=None, white_list=None, out_raw='bench_raw.xlsx', out_summary='bench_summary.xlsx', dump_dir='dump', fLOG=<built-in function print>, verbose=0)#

Runs the benchmark published into the documentation (see Availability of scikit-learn model for runtime onnxruntime1 and Availability of scikit-learn model for runtime python_compiled).

Parameters:

runtime – runtime (python, python_compiled, onnxruntime1, onnxruntime2)
black_list – models to skip, None for none (comma separated list)
white_list – models to benchmark, None for all (comma separated list)
out_raw – all results are saved in that file
out_summary – all results are summarized in that file
dump_dir – folder where to dump intermediate results
fLOG – logging function
verbose – verbosity

Returns:

list of created files

source on GitHub

mlprodict.cli.validate.latency(model, law='normal', size=1, number=10, repeat=10, max_time=0, runtime='onnxruntime', device='cpu', fmt=None, profiling=None, profile_output='profiling.csv')#

Measures the latency of a model (python API).

Parameters:

model – ONNX graph
law – random law used to generate fake inputs
size – batch size, it replaces the first dimension of every input if it is left unknown
number – number of calls to measure
repeat – number of times to repeat the experiment
max_time – if it is > 0, it runs as many time during that period of time
runtime – available runtime
device – device, cpu, cuda:0 or a list of providers `CPUExecutionProvider, CUDAExecutionProvider
fmt – None or csv, it then returns a string formatted like a csv file
profiling – if True, profile the execution of every node, if can be sorted by name or type, the value for this parameter should e in (None, ‘name’, ‘type’)
profile_output – output name for the profiling if profiling is specified

Measures model latency

The command generates random inputs and call many times the model on these inputs. It returns the processing time for one iteration.

Example:

python -m mlprodict latency --model "model.onnx"

<<<

python -m mlprodict latency --help

>>>

usage: latency [-h] [-m MODEL] [--law LAW] [-s SIZE] [-n NUMBER] [-r REPEAT]
               [-ma MAX_TIME] [-ru RUNTIME] [-d DEVICE] [--fmt FMT]
               [-p PROFILING] [-pr PROFILE_OUTPUT]

Measures the latency of a model (python API).

optional arguments:
  -h, --help            show this help message and exit
  -m MODEL, --model MODEL
                        ONNX graph (default: None)
  --law LAW             random law used to generate fake inputs (default:
                        normal)
  -s SIZE, --size SIZE  batch size, it replaces the first dimension of every
                        input if it is left unknown (default: 1)
  -n NUMBER, --number NUMBER
                        number of calls to measure (default: 10)
  -r REPEAT, --repeat REPEAT
                        number of times to repeat the experiment (default: 10)
  -ma MAX_TIME, --max_time MAX_TIME
                        if it is > 0, it runs as many time during that period
                        of time (default: 0)
  -ru RUNTIME, --runtime RUNTIME
                        available runtime (default: onnxruntime)
  -d DEVICE, --device DEVICE
                        device, `cpu`, `cuda:0` or a list of providers
                        `CPUExecutionProvider, CUDAExecutionProvider (default:
                        cpu)
  --fmt FMT             None or `csv`, it then returns a string formatted like
                        a csv file (default: )
  -p PROFILING, --profiling PROFILING
                        if True, profile the execution of every node, if can
                        be sorted by name or type, the value for this
                        parameter should e in `(None, 'name', 'type')`
                        (default: )
  -pr PROFILE_OUTPUT, --profile_output PROFILE_OUTPUT
                        output name for the profiling if profiling is
                        specified (default: profiling.csv)

source on GitHub

mlprodict.cli.validate.validate_runtime(verbose=1, opset_min=-1, opset_max='', check_runtime=True, runtime='python', debug=False, models=None, out_raw='model_onnx_raw.xlsx', out_summary='model_onnx_summary.xlsx', dump_folder=None, dump_all=False, benchmark=False, catch_warnings=True, assume_finite=True, versions=False, skip_models=None, extended_list=True, separate_process=False, time_kwargs=None, n_features=None, fLOG=<built-in function print>, out_graph=None, force_return=False, dtype=None, skip_long_test=False, number=1, repeat=1, time_kwargs_fact='lin', time_limit=4, n_jobs=0)#

Walks through most of scikit-learn operators or model or predictor or transformer, tries to convert them into ONNX and computes the predictions with a specific runtime.

Parameters:

verbose – integer from 0 (None) to 2 (full verbose)
opset_min – tries every conversion from this minimum opset, -1 to get the current opset
opset_max – tries every conversion up to maximum opset, -1 to get the current opset
check_runtime – to check the runtime and not only the conversion
runtime – runtime to check, python, onnxruntime1 to check onnxruntime, onnxruntime2 to check every ONNX node independently with onnxruntime, many runtime can be checked at the same time if the value is a comma separated list
models – comma separated list of models to test or empty string to test them all
skip_models – models to skip
debug – stops whenever an exception is raised, only if separate_process is False
out_raw – output raw results into this file (excel format)
out_summary – output an aggregated view into this file (excel format)
dump_folder – folder where to dump information (pickle) in case of mismatch
dump_all – dumps all models, not only the failing ones
benchmark – run benchmark
catch_warnings – catch warnings
assume_finite – See config_context, If True, validation for finiteness will be skipped, saving time, but leading to potential crashes. If False, validation for finiteness will be performed, avoiding error.
versions – add columns with versions of used packages, numpy, scikit-learn, onnx, onnxruntime, sklearn-onnx
extended_list – extends the list of scikit-learn converters with converters implemented in this module
separate_process – run every model in a separate process, this option must be used to run all model in one row even if one of them is crashing
time_kwargs – a dictionary which defines the number of rows and the parameter number and repeat when benchmarking a model, the value must follow json format
n_features – change the default number of features for a specific problem, it can also be a comma separated list
force_return – forces the function to return the results, used when the results are produces through a separate process
out_graph – image name, to output a graph which summarizes a benchmark in case it was run
dtype – ‘32’ or ‘64’ or None for both, limits the test to one specific number types
skip_long_test – skips tests for high values of N if they seem too long
number – to multiply number values in time_kwargs
repeat – to multiply repeat values in time_kwargs
time_kwargs_fact – to multiply number and repeat in time_kwargs depending on the model (see _multiply_time_kwargs)
time_limit – to stop benchmarking after this limit of time
n_jobs – force the number of jobs to have this value, by default, it is equal to the number of CPU
fLOG – logging function

Validates a runtime against scikit-learn

The command walks through all scikit-learn operators, tries to convert them, checks the predictions, and produces a report.

Example:

python -m mlprodict validate_runtime --models LogisticRegression,LinearRegression

Following example benchmarks models sklearn.ensemble.RandomForestRegressor, sklearn.tree.DecisionTreeRegressor, it compares onnxruntime against scikit-learn for opset 10.

python -m mlprodict validate_runtime -v 1 -o 10 -op 10 -c 1 -r onnxruntime1
       -m RandomForestRegressor,DecisionTreeRegressor -out bench_onnxruntime.xlsx -b 1

Parameter --time_kwargs may be used to reduce or increase bencharmak precisions. The following value tells the function to run a benchmarks with datasets of 1 or 10 number, to repeat a given number of time number predictions in one row. The total time is divided by $number \times repeat`$ . Parameter --time_kwargs_fact may be used to increase these number for some specific models. 'lin' multiplies by 10 number when the model is linear.

-t "{\"1\":{\"number\":10,\"repeat\":10},\"10\":{\"number\":5,\"repeat\":5}}"

The following example dumps every model in the list:

python -m mlprodict validate_runtime --out_raw raw.csv --out_summary sum.csv
       --models LinearRegression,LogisticRegression,DecisionTreeRegressor,DecisionTreeClassifier
       -r python,onnxruntime1 -o 10 -op 10 -v 1 -b 1 -dum 1
       -du model_dump -n 20,100,500 --out_graph benchmark.png --dtype 32

The command line generates a graph produced by function plot_validate_benchmark.

<<<

python -m mlprodict validate_runtime --help

>>>

usage: validate_runtime [-h] [-v VERBOSE] [-o OPSET_MIN] [-op OPSET_MAX]
                        [-c CHECK_RUNTIME] [-r RUNTIME] [-d DEBUG] [-m MODELS]
                        [-ou OUT_RAW] [-out OUT_SUMMARY] [-du DUMP_FOLDER]
                        [-dum DUMP_ALL] [-b BENCHMARK] [-ca CATCH_WARNINGS]
                        [-a ASSUME_FINITE] [-ve VERSIONS] [-s SKIP_MODELS]
                        [-e EXTENDED_LIST] [-se SEPARATE_PROCESS]
                        [-t TIME_KWARGS] [-n N_FEATURES]
                        [--out_graph OUT_GRAPH] [-f FORCE_RETURN] [-dt DTYPE]
                        [-sk SKIP_LONG_TEST] [-nu NUMBER] [-re REPEAT]
                        [-ti TIME_KWARGS_FACT] [-tim TIME_LIMIT] [-n_ N_JOBS]

Walks through most of :epkg:`scikit-learn` operators or model or predictor or
transformer, tries to convert them into `ONNX` and computes the predictions
with a specific runtime.

optional arguments:
  -h, --help            show this help message and exit
  -v VERBOSE, --verbose VERBOSE
                        integer from 0 (None) to 2 (full verbose) (default: 1)
  -o OPSET_MIN, --opset_min OPSET_MIN
                        tries every conversion from this minimum opset, -1 to
                        get the current opset (default: -1)
  -op OPSET_MAX, --opset_max OPSET_MAX
                        tries every conversion up to maximum opset, -1 to get
                        the current opset (default: )
  -c CHECK_RUNTIME, --check_runtime CHECK_RUNTIME
                        to check the runtime and not only the conversion
                        (default: True)
  -r RUNTIME, --runtime RUNTIME
                        runtime to check, python, onnxruntime1 to check
                        `onnxruntime`, onnxruntime2 to check every *ONNX* node
                        independently with onnxruntime, many runtime can be
                        checked at the same time if the value is a comma
                        separated list (default: python)
  -d DEBUG, --debug DEBUG
                        stops whenever an exception is raised, only if
                        *separate_process* is False (default: False)
  -m MODELS, --models MODELS
                        comma separated list of models to test or empty string
                        to test them all (default: )
  -ou OUT_RAW, --out_raw OUT_RAW
                        output raw results into this file (excel format)
                        (default: model_onnx_raw.xlsx)
  -out OUT_SUMMARY, --out_summary OUT_SUMMARY
                        output an aggregated view into this file (excel
                        format) (default: model_onnx_summary.xlsx)
  -du DUMP_FOLDER, --dump_folder DUMP_FOLDER
                        folder where to dump information (pickle) in case of
                        mismatch (default: )
  -dum DUMP_ALL, --dump_all DUMP_ALL
                        dumps all models, not only the failing ones (default:
                        False)
  -b BENCHMARK, --benchmark BENCHMARK
                        run benchmark (default: False)
  -ca CATCH_WARNINGS, --catch_warnings CATCH_WARNINGS
                        catch warnings (default: True)
  -a ASSUME_FINITE, --assume_finite ASSUME_FINITE
                        See `config_context <https://scikit-learn.org/stable/m
                        odules/generated/sklearn.config_context.html>`_, If
                        True, validation for finiteness will be skipped,
                        saving time, but leading to potential crashes. If
                        False, validation for finiteness will be performed,
                        avoiding error. (default: True)
  -ve VERSIONS, --versions VERSIONS
                        add columns with versions of used packages, `numpy`,
                        :epkg:`scikit-learn`, `onnx`, `onnxruntime`,
                        :epkg:`sklearn-onnx` (default: False)
  -s SKIP_MODELS, --skip_models SKIP_MODELS
                        models to skip (default: )
  -e EXTENDED_LIST, --extended_list EXTENDED_LIST
                        extends the list of :epkg:`scikit-learn` converters
                        with converters implemented in this module (default:
                        True)
  -se SEPARATE_PROCESS, --separate_process SEPARATE_PROCESS
                        run every model in a separate process, this option
                        must be used to run all model in one row even if one
                        of them is crashing (default: False)
  -t TIME_KWARGS, --time_kwargs TIME_KWARGS
                        a dictionary which defines the number of rows and the
                        parameter *number* and *repeat* when benchmarking a
                        model, the value must follow `json` format (default: )
  -n N_FEATURES, --n_features N_FEATURES
                        change the default number of features for a specific
                        problem, it can also be a comma separated list
                        (default: )
  --out_graph OUT_GRAPH
                        image name, to output a graph which summarizes a
                        benchmark in case it was run (default: )
  -f FORCE_RETURN, --force_return FORCE_RETURN
                        forces the function to return the results, used when
                        the results are produces through a separate process
                        (default: False)
  -dt DTYPE, --dtype DTYPE
                        '32' or '64' or None for both, limits the test to one
                        specific number types (default: )
  -sk SKIP_LONG_TEST, --skip_long_test SKIP_LONG_TEST
                        skips tests for high values of N if they seem too long
                        (default: False)
  -nu NUMBER, --number NUMBER
                        to multiply number values in *time_kwargs* (default:
                        1)
  -re REPEAT, --repeat REPEAT
                        to multiply repeat values in *time_kwargs* (default:
                        1)
  -ti TIME_KWARGS_FACT, --time_kwargs_fact TIME_KWARGS_FACT
                        to multiply number and repeat in *time_kwargs*
                        depending on the model (see
                        :func:`_multiply_time_kwargs <mlprodict.onnxrt.validat
                        e.validate_helper._multiply_time_kwargs>`) (default:
                        lin)
  -tim TIME_LIMIT, --time_limit TIME_LIMIT
                        to stop benchmarking after this limit of time
                        (default: 4)
  -n_ N_JOBS, --n_jobs N_JOBS
                        force the number of jobs to have this value, by
                        default, it is equal to the number of CPU (default: 0)

source on GitHub

module cli.validate#

Short summary#

Functions#

Documentation#

module `cli.validate`#