Command lines

  1. Automatically creates an asv benchmark

  2. Computes statistics on an ONNX graph

  3. Converts and compares an ONNX file

  4. Converts asv results into csv

  5. Optimises an ONNX graph

  6. Replay a benchmark of stored converted models by validate_runtime

  7. Validate a runtime against scikit-learn

Automatically creates an asv benchmark

The command creates a benchmark based on asv module. It does not run it.

Example:

python -m mlprodict asv_bench --models LogisticRegression,LinearRegression

<<<

python -m mlprodict asv_bench --help

>>>

usage: asv_bench [-h] [-l LOCATION] [-o OPSET_MIN] [-op OPSET_MAX]
                 [-r RUNTIME] [-m MODELS] [-s SKIP_MODELS] [-e EXTENDED_LIST]
                 [-d DIMS] [-n N_FEATURES] [-dt DTYPE] [-v VERBOSE] [-c CLEAN]
                 [-f FLAT] [-co CONF_PARAMS] [-b BUILD] [-a ADD_PYSPY]
                 [--env ENV] [-ma MATRIX]

Creates an `asv` benchmark in a folder but does not run it.

optional arguments:
  -h, --help            show this help message and exit
  -l LOCATION, --location LOCATION
                        location of the benchmark (default: asvsklonnx)
  -o OPSET_MIN, --opset_min OPSET_MIN
                        tries every conversion from this minimum opset, `-1`
                        to get the current opset defined by module onnx
                        (default: -1)
  -op OPSET_MAX, --opset_max OPSET_MAX
                        tries every conversion up to maximum opset, `-1` to
                        get the current opset defined by module onnx (default:
                        )
  -r RUNTIME, --runtime RUNTIME
                        runtime to check, *scikit-learn*, *python*,
                        *python_compiled* compiles the graph structure and is
                        more efficient when the number of observations is
                        small, *onnxruntime1* to check `onnxruntime`,
                        *onnxruntime2* to check every ONNX node independently
                        with onnxruntime, many runtime can be checked at the
                        same time if the value is a comma separated list
                        (default: scikit-learn,python_compiled)
  -m MODELS, --models MODELS
                        list of models to test or empty string to test them
                        all (default: )
  -s SKIP_MODELS, --skip_models SKIP_MODELS
                        models to skip (default: )
  -e EXTENDED_LIST, --extended_list EXTENDED_LIST
                        extends the list of :epkg:`scikit-learn` converters
                        with converters implemented in this module (default:
                        True)
  -d DIMS, --dims DIMS  number of observations to try (default:
                        1,10,100,1000,10000,100000)
  -n N_FEATURES, --n_features N_FEATURES
                        number of features to try (default: 4,20)
  -dt DTYPE, --dtype DTYPE
                        '32' or '64' or None for both, limits the test to one
                        specific number types (default: )
  -v VERBOSE, --verbose VERBOSE
                        integer from 0 (None) to 2 (full verbose) (default: 1)
  -c CLEAN, --clean CLEAN
                        clean the folder first, otherwise overwrites the
                        content (default: True)
  -f FLAT, --flat FLAT  one folder for all files or subfolders (default:
                        False)
  -co CONF_PARAMS, --conf_params CONF_PARAMS
                        to overwrite some of the configuration parameters,
                        format ``name,value;name2,value2`` (default: )
  -b BUILD, --build BUILD
                        location of the outputs (env, html, results) (default:
                        )
  -a ADD_PYSPY, --add_pyspy ADD_PYSPY
                        add an extra folder with code to profile each
                        configuration (default: False)
  --env ENV             default environment or ``same`` to use the current one
                        (default: )
  -ma MATRIX, --matrix MATRIX
                        specifies versions for a module as a json string,
                        example: ``{'onnxruntime': ['1.1.1', '1.1.2']}``, if a
                        package name starts with `'~'`, the package is removed
                        (default: )

(original entry : asv_bench.py:docstring of mlprodict.cli.asv_bench.asv_bench, line 40)

Computes statistics on an ONNX graph

The command computes statistics on an ONNX model.

<<<

python -m mlprodict onnx_stats --help

>>>

usage: onnx_stats [-h] [-n NAME] [-o OPTIM]

Computes statistics on an ONNX model.

optional arguments:
  -h, --help            show this help message and exit
  -n NAME, --name NAME  filename (default: None)
  -o OPTIM, --optim OPTIM
                        computes statistics before an after optimisation was
                        done (default: False)

(original entry : optimize.py:docstring of mlprodict.cli.optimize.onnx_stats, line 6)

Converts and compares an ONNX file

The command converts and validates a scikit-learn model. An example to check the prediction of a logistic regression.

import os
import pickle
import pandas
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from mlprodict.__main__ import main
from mlprodict.cli import convert_validate

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, _ = train_test_split(X, y, random_state=11)
clr = LogisticRegression()
clr.fit(X_train, y_train)

pandas.DataFrame(X_test).to_csv("data.csv", index=False)
with open("model.pkl", "wb") as f:
    pickle.dump(clr, f)

And the command line to check the predictions using a command line.

convert_validate --pkl model.pkl --data data.csv
                 --method predict,predict_proba
                 --name output_label,output_probability
                 --verbose 1

<<<

python -m mlprodict convert_validate --help

>>>

usage: convert_validate [-h] [--pkl PKL] [-d DATA] [-s SCHEMA] [-m METHOD]
                        [-n NAME] [-t TARGET_OPSET] [-o OUTONNX] [-r RUNTIME]
                        [-me METRIC] [-u USE_DOUBLE] [-no NOSHAPE] [-op OPTIM]
                        [-re REWRITE_OPS] [-opt OPTIONS] [-v VERBOSE]
                        [-reg REGISTER]

Converts a model stored in *pkl* file and measure the differences between the
model and the ONNX predictions.

optional arguments:
  -h, --help            show this help message and exit
  --pkl PKL             pickle file (default: None)
  -d DATA, --data DATA  data file, loaded with pandas, converted to a single
                        array, the data is used to guess the schema if
                        *schema* not specified (default: )
  -s SCHEMA, --schema SCHEMA
                        initial type of the model (default: )
  -m METHOD, --method METHOD
                        method to call (default: predict)
  -n NAME, --name NAME  output name (default: Y)
  -t TARGET_OPSET, --target_opset TARGET_OPSET
                        target opset (default: )
  -o OUTONNX, --outonnx OUTONNX
                        produced ONNX model (default: model.onnx)
  -r RUNTIME, --runtime RUNTIME
                        runtime to use to compute predictions, 'python',
                        'python_compiled', 'onnxruntime1' or 'onnxruntime2'
                        (default: python)
  -me METRIC, --metric METRIC
                        the metric 'l1med' is given by function
                        :func:`measure_relative_difference <mlprodict.onnxrt.v
                        alidate.validate_difference.measure_relative_differenc
                        e>` (default: l1med)
  -u USE_DOUBLE, --use_double USE_DOUBLE
                        use double for the runtime if possible, two possible
                        options, ``"float64"`` or ``'switch'``, the first
                        option produces an ONNX file with doubles, the second
                        option loads an ONNX file (float or double) and
                        replaces matrices in ONNX with the matrices coming
                        from the model, this second way is just for testing
                        purposes (default: )
  -no NOSHAPE, --noshape NOSHAPE
                        run the conversion with no shape information (default:
                        False)
  -op OPTIM, --optim OPTIM
                        applies optimisations on the first ONNX graph, use
                        'onnx' to reduce the number of node Identity and
                        redundant subgraphs (default: onnx)
  -re REWRITE_OPS, --rewrite_ops REWRITE_OPS
                        rewrites some converters from skl2onnx (default: True)
  -opt OPTIONS, --options OPTIONS
                        additional options for conversion, dictionary as a
                        string (default: )
  -v VERBOSE, --verbose VERBOSE
                        verbose level (default: 1)
  -reg REGISTER, --register REGISTER
                        registers additional converters implemented by this
                        package (default: True)

(original entry : convert_validate.py:docstring of mlprodict.cli.convert_validate.convert_validate, line 37)

Converts asv results into csv

The command converts asv results into csv.

Example:

python -m mlprodict asv2csv -f <folder> -o result.csv

<<<

python -m mlprodict asv2csv--help

>>>

Command not found: 'asv2csv--help'.

Available commands:

    asv2csv            Converts results produced by :epkg:`asv` into :epkg:`csv`.
    asv_bench          Creates an :epkg:`asv` benchmark in a folder
    benchmark_replay   The command rerun a benchmark if models were stored by
    convert_validate   Converts a model stored in *pkl* file and measure the differences
    onnx_optim         Optimises an ONNX model.
    onnx_stats         Computes statistics on an ONNX model.
    validate_runtime   Walks through most of :epkg:`scikit-learn` operators

(original entry : asv2csv.py:docstring of mlprodict.cli.asv2csv.asv2csv, line 11)

Optimises an ONNX graph

The command optimises an ONNX model.

<<<

python -m mlprodict onnx_optim --help

>>>

usage: onnx_optim [-h] [-n NAME] [-o OUTFILE] [-r RECURSIVE] [-op OPTIONS]
                  [-v VERBOSE]

Optimises an ONNX model.

optional arguments:
  -h, --help            show this help message and exit
  -n NAME, --name NAME  filename (default: None)
  -o OUTFILE, --outfile OUTFILE
                        output filename (default: )
  -r RECURSIVE, --recursive RECURSIVE
                        processes the main graph and the subgraphs (default:
                        True)
  -op OPTIONS, --options OPTIONS
                        options, kind of optimize to do (default: )
  -v VERBOSE, --verbose VERBOSE
                        display statistics before and after the optimisation
                        (default: 0)

(original entry : optimize.py:docstring of mlprodict.cli.optimize.onnx_optim, line 10)

Replay a benchmark of stored converted models by validate_runtime

The command rerun a benchmark if models were stored by command line vaidate_runtime.

Example:

python -m mlprodict benchmark_replay --folder dumped --out bench_results.xlsx

Parameter --time_kwargs may be used to reduce or increase bencharmak precisions. The following value tells the function to run a benchmarks with datasets of 1 or 10 number, to repeat a given number of time number predictions in one row. The total time is divided by number \times repeat. Parameter --time_kwargs_fact may be used to increase these number for some specific models. 'lin' multiplies by 10 number when the model is linear.

-t "{\"1\":{\"number\":10,\"repeat\":10},\"10\":{\"number\":5,\"repeat\":5}}"

<<<

python -m mlprodict benchmark_replay --help

>>>

usage: benchmark_replay [-h] [-f FOLDER] [-r RUNTIME] [-t TIME_KWARGS]
                        [-s SKIP_LONG_TEST] [-ti TIME_KWARGS_FACT]
                        [-tim TIME_LIMIT] [--out OUT] [-v VERBOSE]

The command rerun a benchmark if models were stored by command line
`vaidate_runtime`.

optional arguments:
  -h, --help            show this help message and exit
  -f FOLDER, --folder FOLDER
                        where to find pickled files (default: None)
  -r RUNTIME, --runtime RUNTIME
                        runtimes, comma separated list (default: python)
  -t TIME_KWARGS, --time_kwargs TIME_KWARGS
                        a dictionary which defines the number of rows and the
                        parameter *number* and *repeat* when benchmarking a
                        model, the value must follow `json` format (default: )
  -s SKIP_LONG_TEST, --skip_long_test SKIP_LONG_TEST
                        skips tests for high values of N if they seem too long
                        (default: True)
  -ti TIME_KWARGS_FACT, --time_kwargs_fact TIME_KWARGS_FACT
                        to multiply number and repeat in *time_kwargs*
                        depending on the model (see
                        :func:`_multiply_time_kwargs <mlprodict.onnxrt.validat
                        e.validate_helper._multiply_time_kwargs>`) (default: )
  -tim TIME_LIMIT, --time_limit TIME_LIMIT
                        to stop benchmarking after this limit of time
                        (default: 4)
  --out OUT             output raw results into this file (excel format)
                        (default: )
  -v VERBOSE, --verbose VERBOSE
                        integer from 0 (None) to 2 (full verbose) (default: 1)

(original entry : replay.py:docstring of mlprodict.cli.replay.benchmark_replay, line 19)

Validate a runtime against scikit-learn

The command walks through all scikit-learn operators, tries to convert them, checks the predictions, and produces a report.

Example:

python -m mlprodict validate_runtime --models LogisticRegression,LinearRegression

Following example benchmarks models sklearn.ensemble.RandomForestRegressor, sklearn.tree.DecisionTreeRegressor, it compares onnxruntime against scikit-learn for opset 10.

python -m mlprodict validate_runtime -v 1 -o 10 -op 10 -c 1 -r onnxruntime1
       -m RandomForestRegressor,DecisionTreeRegressor -out bench_onnxruntime.xlsx -b 1

Parameter --time_kwargs may be used to reduce or increase bencharmak precisions. The following value tells the function to run a benchmarks with datasets of 1 or 10 number, to repeat a given number of time number predictions in one row. The total time is divided by number \times repeat. Parameter --time_kwargs_fact may be used to increase these number for some specific models. 'lin' multiplies by 10 number when the model is linear.

-t "{\"1\":{\"number\":10,\"repeat\":10},\"10\":{\"number\":5,\"repeat\":5}}"

The following example dumps every model in the list:

python -m mlprodict validate_runtime --out_raw raw.csv --out_summary sum.csv
       --models LinearRegression,LogisticRegression,DecisionTreeRegressor,DecisionTreeClassifier
       -r python,onnxruntime1 -o 10 -op 10 -v 1 -b 1 -dum 1
       -du model_dump -n 20,100,500 --out_graph benchmark.png --dtype 32

The command line generates a graph produced by function plot_validate_benchmark.

<<<

python -m mlprodict validate_runtime --help

>>>

usage: validate_runtime [-h] [-v VERBOSE] [-o OPSET_MIN] [-op OPSET_MAX]
                        [-c CHECK_RUNTIME] [-r RUNTIME] [-d DEBUG] [-m MODELS]
                        [-ou OUT_RAW] [-out OUT_SUMMARY] [-du DUMP_FOLDER]
                        [-dum DUMP_ALL] [-b BENCHMARK] [-ca CATCH_WARNINGS]
                        [-a ASSUME_FINITE] [-ve VERSIONS] [-s SKIP_MODELS]
                        [-e EXTENDED_LIST] [-se SEPARATE_PROCESS]
                        [-t TIME_KWARGS] [-n N_FEATURES]
                        [--out_graph OUT_GRAPH] [-f FORCE_RETURN] [-dt DTYPE]
                        [-sk SKIP_LONG_TEST] [-nu NUMBER] [-re REPEAT]
                        [-ti TIME_KWARGS_FACT] [-tim TIME_LIMIT] [-n_ N_JOBS]

Walks through most of :epkg:`scikit-learn` operators or model or predictor or
transformer, tries to convert them into `ONNX` and computes the predictions
with a specific runtime.

optional arguments:
  -h, --help            show this help message and exit
  -v VERBOSE, --verbose VERBOSE
                        integer from 0 (None) to 2 (full verbose) (default: 1)
  -o OPSET_MIN, --opset_min OPSET_MIN
                        tries every conversion from this minimum opset, -1 to
                        get the current opset (default: -1)
  -op OPSET_MAX, --opset_max OPSET_MAX
                        tries every conversion up to maximum opset, -1 to get
                        the current opset (default: )
  -c CHECK_RUNTIME, --check_runtime CHECK_RUNTIME
                        to check the runtime and not only the conversion
                        (default: True)
  -r RUNTIME, --runtime RUNTIME
                        runtime to check, python, onnxruntime1 to check
                        `onnxruntime`, onnxruntime2 to check every *ONNX* node
                        independently with onnxruntime, many runtime can be
                        checked at the same time if the value is a comma
                        separated list (default: python)
  -d DEBUG, --debug DEBUG
                        stops whenever an exception is raised, only if
                        *separate_process* is False (default: False)
  -m MODELS, --models MODELS
                        comma separated list of models to test or empty string
                        to test them all (default: )
  -ou OUT_RAW, --out_raw OUT_RAW
                        output raw results into this file (excel format)
                        (default: model_onnx_raw.xlsx)
  -out OUT_SUMMARY, --out_summary OUT_SUMMARY
                        output an aggregated view into this file (excel
                        format) (default: model_onnx_summary.xlsx)
  -du DUMP_FOLDER, --dump_folder DUMP_FOLDER
                        folder where to dump information (pickle) in case of
                        mismatch (default: )
  -dum DUMP_ALL, --dump_all DUMP_ALL
                        dumps all models, not only the failing ones (default:
                        False)
  -b BENCHMARK, --benchmark BENCHMARK
                        run benchmark (default: False)
  -ca CATCH_WARNINGS, --catch_warnings CATCH_WARNINGS
                        catch warnings (default: True)
  -a ASSUME_FINITE, --assume_finite ASSUME_FINITE
                        See `config_context <https://scikit-learn.org/stable/m
                        odules/generated/sklearn.config_context.html>`_, If
                        True, validation for finiteness will be skipped,
                        saving time, but leading to potential crashes. If
                        False, validation for finiteness will be performed,
                        avoiding error. (default: True)
  -ve VERSIONS, --versions VERSIONS
                        add columns with versions of used packages, `numpy`,
                        :epkg:`scikit-learn`, `onnx`, `onnxruntime`,
                        :epkg:`sklearn-onnx` (default: False)
  -s SKIP_MODELS, --skip_models SKIP_MODELS
                        models to skip (default: )
  -e EXTENDED_LIST, --extended_list EXTENDED_LIST
                        extends the list of :epkg:`scikit-learn` converters
                        with converters implemented in this module (default:
                        True)
  -se SEPARATE_PROCESS, --separate_process SEPARATE_PROCESS
                        run every model in a separate process, this option
                        must be used to run all model in one row even if one
                        of them is crashing (default: False)
  -t TIME_KWARGS, --time_kwargs TIME_KWARGS
                        a dictionary which defines the number of rows and the
                        parameter *number* and *repeat* when benchmarking a
                        model, the value must follow `json` format (default: )
  -n N_FEATURES, --n_features N_FEATURES
                        change the default number of features for a specific
                        problem, it can also be a comma separated list
                        (default: )
  --out_graph OUT_GRAPH
                        image name, to output a graph which summarizes a
                        benchmark in case it was run (default: )
  -f FORCE_RETURN, --force_return FORCE_RETURN
                        forces the function to return the results, used when
                        the results are produces through a separate process
                        (default: False)
  -dt DTYPE, --dtype DTYPE
                        '32' or '64' or None for both, limits the test to one
                        specific number types (default: )
  -sk SKIP_LONG_TEST, --skip_long_test SKIP_LONG_TEST
                        skips tests for high values of N if they seem too long
                        (default: False)
  -nu NUMBER, --number NUMBER
                        to multiply number values in *time_kwargs* (default:
                        1)
  -re REPEAT, --repeat REPEAT
                        to multiply repeat values in *time_kwargs* (default:
                        1)
  -ti TIME_KWARGS_FACT, --time_kwargs_fact TIME_KWARGS_FACT
                        to multiply number and repeat in *time_kwargs*
                        depending on the model (see
                        :func:`_multiply_time_kwargs <mlprodict.onnxrt.validat
                        e.validate_helper._multiply_time_kwargs>`) (default:
                        lin)
  -tim TIME_LIMIT, --time_limit TIME_LIMIT
                        to stop benchmarking after this limit of time
                        (default: 4)
  -n_ N_JOBS, --n_jobs N_JOBS
                        force the number of jobs to have this value, by
                        default, it is equal to the number of CPU (default: 0)

(original entry : validate.py:docstring of mlprodict.cli.validate.validate_runtime, line 66)