.. _f-optokenizer: module ``onnxrt.ops_cpu.op_tokenizer`` ====================================== .. inheritance-diagram:: mlprodict.onnxrt.ops_cpu.op_tokenizer Short summary +++++++++++++ module ``mlprodict.onnxrt.ops_cpu.op_tokenizer`` Runtime operator. :githublink:`%|py|7` Classes +++++++ +----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+ | class | truncated documentation | +==================================================================================+=====================================================================================================+ | :class:`Tokenizer ` | See :epkg:`Tokenizer`. | +----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+ | :class:`TokenizerSchema ` | Defines a schema for operators added in this package such as :class:`TreeEnsembleClassifierDouble`. | +----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+ Properties ++++++++++ +-------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+ | property | truncated documentation | +=======================================================================================================+=======================================================================================================================+ | :meth:`args_default ` | Returns the list of arguments as well as the list of parameters with the default values (close to the signature). ... | +-------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+ | :meth:`args_default_modified ` | Returns the list of modified parameters. | +-------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+ | :meth:`args_mandatory ` | Returns the list of optional arguments. | +-------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+ | :meth:`args_optional ` | Returns the list of optional arguments. | +-------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+ | :meth:`atts_value ` | Returns all parameters in a dictionary. | +-------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+ Methods +++++++ +------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+ | method | truncated documentation | +========================================================================================================================+==========================================================================+ | :py:meth:`__init__ ` | | +------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+ | :py:meth:`__init__ ` | | +------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+ | :py:meth:`_find_custom_operator_schema ` | | +------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+ | :py:meth:`_run ` | | +------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+ | :py:meth:`_run_char_tokenization ` | Tokenizes y charaters. | +------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+ | :py:meth:`_run_regex_tokenization ` | Tokenizes using separators. The function should use a trie to find text. | +------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+ | :py:meth:`_run_sep_tokenization ` | Tokenizes using separators. The function should use a trie to find text. | +------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+ | :py:meth:`_run_tokenization ` | Tokenizes a char level. | +------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+ Documentation +++++++++++++ .. automodule:: mlprodict.onnxrt.ops_cpu.op_tokenizer :members: :special-members: __init__ :show-inheritance: