module onnx_conv.onnx_ops.onnx_tokenizer

Short summary

module mlprodict.onnx_conv.onnx_ops.onnx_tokenizer

Custom operator Tokenizer.

source on GitHub

Classes

class	truncated documentation
OnnxTokenizer_1	Defines a custom operator not defined by ONNX specifications but in onnxruntime.
OnnxTokenizer_1	Defines a custom operator not defined by ONNX specifications but in onnxruntime.

Properties

property	truncated documentation
onnx_prefix
onnx_prefix
outputs	Returns the outputs of the node.
outputs	Returns the outputs of the node.

Methods

method	truncated documentation
__init__
__init__

Documentation

Custom operator Tokenizer.

source on GitHub

mlprodict.onnx_conv.onnx_ops.onnx_tokenizer.OnnxTokenizer

alias of OnnxTokenizer_1

class mlprodict.onnx_conv.onnx_ops.onnx_tokenizer.OnnxTokenizer_1(text, mark=0, mincharnum=1, pad_value='#', separators=None, tokenexp='[a-zA-Z0-9_]+', stopwords=None, op_version=None, **kwargs)

Bases: OnnxOperator

Defines a custom operator not defined by ONNX specifications but in onnxruntime.

source on GitHub

Parameters:

• text – array or OnnxOperatorMixin

• mark – see Tokenizer

• pad_value – see Tokenizer

• separators – see Tokenizer

• tokenexp – see Tokenizer

• stopwords – list of stopwords, addition to Tokenizer

• op_version – opset version

• kwargs – additional parameter

source on GitHub

__init__(text, mark=0, mincharnum=1, pad_value='#', separators=None, tokenexp='[a-zA-Z0-9_]+', stopwords=None, op_version=None, **kwargs)

Parameters:

• text – array or OnnxOperatorMixin

• mark – see Tokenizer

• pad_value – see Tokenizer

• separators – see Tokenizer

• tokenexp – see Tokenizer

• stopwords – list of stopwords, addition to Tokenizer

• op_version – opset version

• kwargs – additional parameter

source on GitHub