module cli.validate

Short summary

module mlprodict.cli.validate

Command line about validation of prediction runtime.

source on GitHub

Functions

function

truncated documentation

_finalize

_validate_runtime_dict

_validate_runtime_separate_process

validate_runtime

Walks through most of scikit-learn operators or model or predictor or transformer, tries to convert them …

Documentation

Command line about validation of prediction runtime.

source on GitHub

mlprodict.cli.validate._finalize(rows, out_raw, out_summary, verbose, models, out_graph, fLOG)
mlprodict.cli.validate._validate_runtime_dict(kwargs)
mlprodict.cli.validate._validate_runtime_separate_process(**kwargs)
mlprodict.cli.validate.validate_runtime(verbose=1, opset_min=-1, opset_max='', check_runtime=True, runtime='python', debug=False, models=None, out_raw='model_onnx_raw.xlsx', out_summary='model_onnx_summary.xlsx', dump_folder=None, dump_all=False, benchmark=False, catch_warnings=True, assume_finite=True, versions=False, skip_models=None, extended_list=True, separate_process=False, time_kwargs=None, n_features=None, fLOG=<built-in function print>, out_graph=None, force_return=False, dtype=None, skip_long_test=False, number=1, repeat=1, time_kwargs_fact='lin', time_limit=4, n_jobs=0)

Walks through most of scikit-learn operators or model or predictor or transformer, tries to convert them into ONNX and computes the predictions with a specific runtime.

Parameters
  • verbose – integer from 0 (None) to 2 (full verbose)

  • opset_min – tries every conversion from this minimum opset, -1 to get the current opset

  • opset_max – tries every conversion up to maximum opset, -1 to get the current opset

  • check_runtime – to check the runtime and not only the conversion

  • runtime – runtime to check, python, onnxruntime1 to check onnxruntime, onnxruntime2 to check every ONNX node independently with onnxruntime, many runtime can be checked at the same time if the value is a comma separated list

  • models – comma separated list of models to test or empty string to test them all

  • skip_models – models to skip

  • debug – stops whenever an exception is raised, only if separate_process is False

  • out_raw – output raw results into this file (excel format)

  • out_summary – output an aggregated view into this file (excel format)

  • dump_folder – folder where to dump information (pickle) in case of mismatch

  • dump_all – dumps all models, not only the failing ones

  • benchmark – run benchmark

  • catch_warnings – catch warnings

  • assume_finite – See config_context, If True, validation for finiteness will be skipped, saving time, but leading to potential crashes. If False, validation for finiteness will be performed, avoiding error.

  • versions – add columns with versions of used packages, numpy, scikit-learn, onnx, onnxruntime, sklearn-onnx

  • extended_list – extends the list of scikit-learn converters with converters implemented in this module

  • separate_process – run every model in a separate process, this option must be used to run all model in one row even if one of them is crashing

  • time_kwargs – a dictionary which defines the number of rows and the parameter number and repeat when benchmarking a model, the value must follow json format

  • n_features – change the default number of features for a specific problem, it can also be a comma separated list

  • force_return – forces the function to return the results, used when the results are produces through a separate process

  • out_graph – image name, to output a graph which summarizes a benchmark in case it was run

  • dtype – ‘32’ or ‘64’ or None for both, limits the test to one specific number types

  • skip_long_test – skips tests for high values of N if they seem too long

  • number – to multiply number values in time_kwargs

  • repeat – to multiply repeat values in time_kwargs

  • time_kwargs_fact – to multiply number and repeat in time_kwargs depending on the model (see _multiply_time_kwargs)

  • time_limit – to stop benchmarking after this limit of time

  • n_jobs – force the number of jobs to have this value, by default, it is equal to the number of CPU

  • fLOG – logging function

Validates a runtime against scikit-learn

The command walks through all scikit-learn operators, tries to convert them, checks the predictions, and produces a report.

Example:

python -m mlprodict validate_runtime --models LogisticRegression,LinearRegression

Following example benchmarks models sklearn.ensemble.RandomForestRegressor, sklearn.tree.DecisionTreeRegressor, it compares onnxruntime against scikit-learn for opset 10.

python -m mlprodict validate_runtime -v 1 -o 10 -op 10 -c 1 -r onnxruntime1
       -m RandomForestRegressor,DecisionTreeRegressor -out bench_onnxruntime.xlsx -b 1

Parameter --time_kwargs may be used to reduce or increase bencharmak precisions. The following value tells the function to run a benchmarks with datasets of 1 or 10 number, to repeat a given number of time number predictions in one row. The total time is divided by number \times repeat. Parameter --time_kwargs_fact may be used to increase these number for some specific models. 'lin' multiplies by 10 number when the model is linear.

-t "{\"1\":{\"number\":10,\"repeat\":10},\"10\":{\"number\":5,\"repeat\":5}}"

The following example dumps every model in the list:

python -m mlprodict validate_runtime --out_raw raw.csv --out_summary sum.csv
       --models LinearRegression,LogisticRegression,DecisionTreeRegressor,DecisionTreeClassifier
       -r python,onnxruntime1 -o 10 -op 10 -v 1 -b 1 -dum 1
       -du model_dump -n 20,100,500 --out_graph benchmark.png --dtype 32

The command line generates a graph produced by function plot_validate_benchmark.

<<<

python -m mlprodict validate_runtime --help

>>>

usage: validate_runtime [-h] [-v VERBOSE] [-o OPSET_MIN] [-op OPSET_MAX]
                        [-c CHECK_RUNTIME] [-r RUNTIME] [-d DEBUG] [-m MODELS]
                        [-ou OUT_RAW] [-out OUT_SUMMARY] [-du DUMP_FOLDER]
                        [-dum DUMP_ALL] [-b BENCHMARK] [-ca CATCH_WARNINGS]
                        [-a ASSUME_FINITE] [-ve VERSIONS] [-s SKIP_MODELS]
                        [-e EXTENDED_LIST] [-se SEPARATE_PROCESS]
                        [-t TIME_KWARGS] [-n N_FEATURES]
                        [--out_graph OUT_GRAPH] [-f FORCE_RETURN] [-dt DTYPE]
                        [-sk SKIP_LONG_TEST] [-nu NUMBER] [-re REPEAT]
                        [-ti TIME_KWARGS_FACT] [-tim TIME_LIMIT] [-n_ N_JOBS]

Walks through most of :epkg:`scikit-learn` operators or model or predictor or
transformer, tries to convert them into `ONNX` and computes the predictions
with a specific runtime.

optional arguments:
  -h, --help            show this help message and exit
  -v VERBOSE, --verbose VERBOSE
                        integer from 0 (None) to 2 (full verbose) (default: 1)
  -o OPSET_MIN, --opset_min OPSET_MIN
                        tries every conversion from this minimum opset, -1 to
                        get the current opset (default: -1)
  -op OPSET_MAX, --opset_max OPSET_MAX
                        tries every conversion up to maximum opset, -1 to get
                        the current opset (default: )
  -c CHECK_RUNTIME, --check_runtime CHECK_RUNTIME
                        to check the runtime and not only the conversion
                        (default: True)
  -r RUNTIME, --runtime RUNTIME
                        runtime to check, python, onnxruntime1 to check
                        `onnxruntime`, onnxruntime2 to check every *ONNX* node
                        independently with onnxruntime, many runtime can be
                        checked at the same time if the value is a comma
                        separated list (default: python)
  -d DEBUG, --debug DEBUG
                        stops whenever an exception is raised, only if
                        *separate_process* is False (default: False)
  -m MODELS, --models MODELS
                        comma separated list of models to test or empty string
                        to test them all (default: )
  -ou OUT_RAW, --out_raw OUT_RAW
                        output raw results into this file (excel format)
                        (default: model_onnx_raw.xlsx)
  -out OUT_SUMMARY, --out_summary OUT_SUMMARY
                        output an aggregated view into this file (excel
                        format) (default: model_onnx_summary.xlsx)
  -du DUMP_FOLDER, --dump_folder DUMP_FOLDER
                        folder where to dump information (pickle) in case of
                        mismatch (default: )
  -dum DUMP_ALL, --dump_all DUMP_ALL
                        dumps all models, not only the failing ones (default:
                        False)
  -b BENCHMARK, --benchmark BENCHMARK
                        run benchmark (default: False)
  -ca CATCH_WARNINGS, --catch_warnings CATCH_WARNINGS
                        catch warnings (default: True)
  -a ASSUME_FINITE, --assume_finite ASSUME_FINITE
                        See `config_context <https://scikit-learn.org/stable/m
                        odules/generated/sklearn.config_context.html>`_, If
                        True, validation for finiteness will be skipped,
                        saving time, but leading to potential crashes. If
                        False, validation for finiteness will be performed,
                        avoiding error. (default: True)
  -ve VERSIONS, --versions VERSIONS
                        add columns with versions of used packages, `numpy`,
                        :epkg:`scikit-learn`, `onnx`, `onnxruntime`,
                        :epkg:`sklearn-onnx` (default: False)
  -s SKIP_MODELS, --skip_models SKIP_MODELS
                        models to skip (default: )
  -e EXTENDED_LIST, --extended_list EXTENDED_LIST
                        extends the list of :epkg:`scikit-learn` converters
                        with converters implemented in this module (default:
                        True)
  -se SEPARATE_PROCESS, --separate_process SEPARATE_PROCESS
                        run every model in a separate process, this option
                        must be used to run all model in one row even if one
                        of them is crashing (default: False)
  -t TIME_KWARGS, --time_kwargs TIME_KWARGS
                        a dictionary which defines the number of rows and the
                        parameter *number* and *repeat* when benchmarking a
                        model, the value must follow `json` format (default: )
  -n N_FEATURES, --n_features N_FEATURES
                        change the default number of features for a specific
                        problem, it can also be a comma separated list
                        (default: )
  --out_graph OUT_GRAPH
                        image name, to output a graph which summarizes a
                        benchmark in case it was run (default: )
  -f FORCE_RETURN, --force_return FORCE_RETURN
                        forces the function to return the results, used when
                        the results are produces through a separate process
                        (default: False)
  -dt DTYPE, --dtype DTYPE
                        '32' or '64' or None for both, limits the test to one
                        specific number types (default: )
  -sk SKIP_LONG_TEST, --skip_long_test SKIP_LONG_TEST
                        skips tests for high values of N if they seem too long
                        (default: False)
  -nu NUMBER, --number NUMBER
                        to multiply number values in *time_kwargs* (default:
                        1)
  -re REPEAT, --repeat REPEAT
                        to multiply repeat values in *time_kwargs* (default:
                        1)
  -ti TIME_KWARGS_FACT, --time_kwargs_fact TIME_KWARGS_FACT
                        to multiply number and repeat in *time_kwargs*
                        depending on the model (see
                        :func:`_multiply_time_kwargs <mlprodict.onnxrt.validat
                        e.validate_helper._multiply_time_kwargs>`) (default:
                        lin)
  -tim TIME_LIMIT, --time_limit TIME_LIMIT
                        to stop benchmarking after this limit of time
                        (default: 4)
  -n_ N_JOBS, --n_jobs N_JOBS
                        force the number of jobs to have this value, by
                        default, it is equal to the number of CPU (default: 0)

source on GitHub