module testing.test_utils.tests_helper

Short summary

module mlprodict.testing.test_utils.tests_helper

Inspired from sklearn-onnx, handles two backends.

source on GitHub

Functions

function	truncated documentation
_has_decision_function
_has_predict_proba
_has_transform_model
_raw_score_binary_classification
_save_model_dump
binary_array_to_string	Used to compare decision path.
compute_benchmark	Compares the processing time several functions.
convert_model	Runs the appropriate conversion method.
dump_binary_classification	Trains and dumps a model for a binary classification problem. The function trains a model and calls dump_data_and_model(). …
dump_data_and_model	Saves data with pickle, saves the model with pickle and onnx, runs and saves the predictions for the given model. …
dump_multilabel_classification	Trains and dumps a model for a binary classification problem. The function trains a model and calls dump_data_and_model(). …
dump_multiple_classification	Trains and dumps a model for a binary classification problem. The function trains a model and calls dump_data_and_model(). …
dump_multiple_regression	Trains and dumps a model for a multi regression problem. The function trains a model and calls dump_data_and_model(). …
dump_one_class_classification	Trains and dumps a model for a One Class outlier problem. The function trains a model and calls dump_data_and_model(). …
dump_single_regression	Trains and dumps a model for a regression problem. The function trains a model and calls dump_data_and_model(). …
fit_classification_model	Fits a classification model.
fit_classification_model_simple	Fits a classification model.
fit_multilabel_classification_model	Fits a classification model.
fit_regression_model	Fits a regression model.
timeexec	Measures the time for a given expression.
timeit_repeat	Returns a series of repeat time measures for number executions of code assuming fct is a function.

Documentation

Inspired from sklearn-onnx, handles two backends.

source on GitHub

mlprodict.testing.test_utils.tests_helper._has_decision_function(model)

mlprodict.testing.test_utils.tests_helper._has_predict_proba(model)

mlprodict.testing.test_utils.tests_helper._has_transform_model(model)

mlprodict.testing.test_utils.tests_helper._raw_score_binary_classification(model, X)

mlprodict.testing.test_utils.tests_helper._save_model_dump(model, folder, basename, names)

mlprodict.testing.test_utils.tests_helper.binary_array_to_string(mat)

Used to compare decision path.

source on GitHub

mlprodict.testing.test_utils.tests_helper.compute_benchmark(fcts, number=10, repeat=100)

Compares the processing time several functions.

Parameters:

• fcts – dictionary {'name': fct}

• number – number of time to run the expression (and then divide by this number to get an average)

• repeat – number of times to repeat the computation of the above average

Returns:

list of [{‘name’: name, ‘time’: …}]

source on GitHub

mlprodict.testing.test_utils.tests_helper.convert_model(model, name, input_types)

Runs the appropriate conversion method.

Parameters:

• model – model, scikit-learn, keras, or coremltools object

• name – model name

• input_types – input types

Returns:

onnx model

source on GitHub

mlprodict.testing.test_utils.tests_helper.dump_binary_classification(model, suffix='', folder=None, allow_failure=None, comparable_outputs=None, verbose=False, label_string=False, benchmark=False, methods=None, nrows=None)

Trains and dumps a model for a binary classification problem. The function trains a model and calls dump_data_and_model.

Every created filename will follow the pattern: <folder>/<prefix><task><classifier-name><suffix>.<data|expected|model|onnx>.<pkl|onnx>.

source on GitHub

mlprodict.testing.test_utils.tests_helper.dump_data_and_model(data, model, onnx_model=None, basename='model', folder=None, inputs=None, backend=('python', 'onnxruntime'), context=None, allow_failure=None, methods=None, dump_error_log=None, benchmark=None, comparable_outputs=None, intermediate_steps=False, fail_evenif_notimplemented=False, verbose=False, classes=None, check_error=None, disable_optimisation=False)

Saves data with pickle, saves the model with pickle and onnx, runs and saves the predictions for the given model. This function is used to test a backend (runtime) for onnx.

Parameters:

• data – any kind of data

• model – any model

• onnx_model – onnx model or None to use an onnx converters to convert it only if the model accepts one float vector

• basename – three files are writen <basename>.data.pkl, <basename>.model.pkl, <basename>.model.onnx

• folder – files are written in this folder, it is created if it does not exist, if folder is None, it looks first in environment variable ONNXTESTDUMP, otherwise, it is placed into 'temp_dump'.

• inputs – standard type or specific one if specified, only used is parameter onnx is None

• backend – backend used to compare expected output and runtime output. Two options are currently supported: None for no test, ‘onnxruntime’ to use module onnxruntime, python to use the python runtiume.

• context – used if the model contains a custom operator such as a custom Keras function…

• allow_failure – None to raise an exception if comparison fails for the backends, otherwise a string which is then evaluated to check whether or not the test can fail, example: "StrictVersion(onnx.__version__) < StrictVersion('1.3.0')"

• dump_error_log –

  if True, dumps any error message in a file <basename>.err, if it is None, it checks the environment

  variable ONNXTESTDUMPERROR

• benchmark – if True, runs a benchmark and stores the results into a file <basename>.bench, if None, it checks the environment variable ONNXTESTBENCHMARK

• verbose – additional information

• methods – ONNX may produce one or several results, each of them is equivalent to the output of a method from the model class, this parameter defines which methods is equivalent to ONNX outputs. If not specified, it falls back into a default behaviour implemented for classifiers, regressors, clustering.

• comparable_outputs – compares only these outputs

• intermediate_steps – displays intermediate steps in case of an error

• fail_evenif_notimplemented – the test is considered as failing even if the error is due to onnxuntime missing the implementation of a new operator defiend in ONNX.

• classes – classes names (only for classifier, mandatory if option ‘nocl’ is used)

• check_error – do not raise an exception if the error message contains this text

• disable_optimisation – disable all optimisations onnxruntime could do

Returns:

the created files

Some convention for the name, Bin for a binary classifier, Mcl for a multiclass classifier, Reg for a regressor, MRg for a multi-regressor. The name can contain some flags. Expected outputs refer to the outputs computed with the original library, computed outputs refer to the outputs computed with a ONNX runtime.

• -CannotLoad: the model can be converted but the runtime cannot load it

• -Dec3: compares expected and computed outputs up to 3 decimals (5 by default)

• -Dec4: compares expected and computed outputs up to 4 decimals (5 by default)

• -NoProb: The original models computed probabilites for two classes size=(N, 2) but the runtime produces a vector of size N, the test will compare the second column to the column

• -Out0: only compares the first output on both sides

• -Reshape: merges all outputs into one single vector and resizes it before comparing

• -SkipDim1: before comparing expected and computed output, arrays with a shape like (2, 1, 2) becomes (2, 2)

• -SklCol: scikit-learn operator applies on a column and not a matrix

If the backend is not None, the function either raises an exception if the comparison between the expected outputs and the backend outputs fails or it saves the backend output and adds it to the results.

source on GitHub

mlprodict.testing.test_utils.tests_helper.dump_multilabel_classification(model, suffix='', folder=None, allow_failure=None, verbose=False, label_string=False, first_class=0, comparable_outputs=None, benchmark=False, backend=('python', 'onnxruntime'))

Trains and dumps a model for a binary classification problem. The function trains a model and calls dump_data_and_model.

Every created filename will follow the pattern: <folder>/<prefix><task><classifier-name><suffix>.<data|expected|model|onnx>.<pkl|onnx>.

source on GitHub

mlprodict.testing.test_utils.tests_helper.dump_multiple_classification(model, suffix='', folder=None, allow_failure=None, verbose=False, label_string=False, first_class=0, comparable_outputs=None, benchmark=False, methods=None)

Trains and dumps a model for a binary classification problem. The function trains a model and calls dump_data_and_model.

Every created filename will follow the pattern: <folder>/<prefix><task><classifier-name><suffix>.<data|expected|model|onnx>.<pkl|onnx>.

source on GitHub

mlprodict.testing.test_utils.tests_helper.dump_multiple_regression(model, suffix='', folder=None, allow_failure=None, comparable_outputs=None, verbose=False, benchmark=False)

Trains and dumps a model for a multi regression problem. The function trains a model and calls dump_data_and_model.

Every created filename will follow the pattern: <folder>/<prefix><task><classifier-name><suffix>.<data|expected|model|onnx>.<pkl|onnx>.

source on GitHub

mlprodict.testing.test_utils.tests_helper.dump_one_class_classification(model, suffix='', folder=None, allow_failure=None, comparable_outputs=None, verbose=False, benchmark=False, methods=None)

Trains and dumps a model for a One Class outlier problem. The function trains a model and calls dump_data_and_model.

Every created filename will follow the pattern: <folder>/<prefix><task><classifier-name><suffix>.<data|expected|model|onnx>.<pkl|onnx>.

source on GitHub

mlprodict.testing.test_utils.tests_helper.dump_single_regression(model, suffix='', folder=None, allow_failure=None, comparable_outputs=None, benchmark=False)

Trains and dumps a model for a regression problem. The function trains a model and calls dump_data_and_model.

Every created filename will follow the pattern: <folder>/<prefix><task><classifier-name><suffix>.<data|expected|model|onnx>.<pkl|onnx>.

source on GitHub

mlprodict.testing.test_utils.tests_helper.fit_classification_model(model, n_classes, is_int=False, pos_features=False, label_string=False, random_state=42, is_bool=False, n_features=20)

Fits a classification model.

source on GitHub

mlprodict.testing.test_utils.tests_helper.fit_classification_model_simple(model, n_classes, is_int=False, pos_features=False)

Fits a classification model.

source on GitHub

mlprodict.testing.test_utils.tests_helper.fit_multilabel_classification_model(model, n_classes=5, n_labels=2, n_samples=400, n_features=20, is_int=False)

Fits a classification model.

source on GitHub

mlprodict.testing.test_utils.tests_helper.fit_regression_model(model, is_int=False, n_targets=1, is_bool=False, factor=1.0, n_features=10, n_samples=500, n_informative=10)

Fits a regression model.

source on GitHub

mlprodict.testing.test_utils.tests_helper.timeexec(fct, number, repeat)

Measures the time for a given expression.

Parameters:

• fct – function to measure (as a string)

• number – number of time to run the expression (and then divide by this number to get an average)

• repeat – number of times to repeat the computation of the above average

Returns:

dictionary

source on GitHub

mlprodict.testing.test_utils.tests_helper.timeit_repeat(fct, number, repeat)

Returns a series of repeat time measures for number executions of code assuming fct is a function.

source on GitHub