.. _f-onnxtokenizerz:

module ``sklapi.onnx_tokenizer``
================================





.. inheritance-diagram:: mlprodict.sklapi.onnx_tokenizer


Short summary
+++++++++++++

module ``mlprodict.sklapi.onnx_tokenizer``

Wrapper tokenizrs implemented in :epkg:`onnxruntime-extensions`.


:githublink:`%|py|6`




Classes
+++++++

+----------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| class                                                                                                          | truncated documentation                                                                                                                             |
+================================================================================================================+=====================================================================================================================================================+
| :class:`GPT2TokenizerTransformer <mlprodict.sklapi.onnx_tokenizer.GPT2TokenizerTransformer>`                   | Wraps `GPT2Tokenizer <https://github.com/microsoft/onnxruntime-extensions/blob/ main/docs/custom_text_ops.md#gpt2tokenizer>`_ ...                   |
+----------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| :class:`SentencePieceTokenizerTransformer <mlprodict.sklapi.onnx_tokenizer.SentencePieceTokenizerTransformer>` | Wraps `SentencePieceTokenizer <https://github.com/microsoft/onnxruntime-extensions/blob/ main/docs/custom_text_ops.md#sentencepiecetokenizer>`_ ... |
+----------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| :class:`TokenizerTransformerBase <mlprodict.sklapi.onnx_tokenizer.TokenizerTransformerBase>`                   | Base class for :class:`SentencePieceTokenizerTransformer` and :class:`GPT2TokenizerTransformer`.                                                    |
+----------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+






Properties
++++++++++

+--------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| property                                                                                               | truncated documentation                                                                                          |
+========================================================================================================+==================================================================================================================+
| :py:meth:`_repr_html_ <mlprodict.sklapi.onnx_tokenizer.GPT2TokenizerTransformer._repr_html_>`          | HTML representation of estimator. This is redundant with the logic of `_repr_mimebundle_`. The latter should ... |
+--------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| :py:meth:`_repr_html_ <mlprodict.sklapi.onnx_tokenizer.SentencePieceTokenizerTransformer._repr_html_>` | HTML representation of estimator. This is redundant with the logic of `_repr_mimebundle_`. The latter should ... |
+--------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| :py:meth:`_repr_html_ <mlprodict.sklapi.onnx_tokenizer.TokenizerTransformerBase._repr_html_>`          | HTML representation of estimator. This is redundant with the logic of `_repr_mimebundle_`. The latter should ... |
+--------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+




Static Methods
++++++++++++++

+------------------------------------------------------------------------------------------------------------+-------------------------+
| staticmethod                                                                                               | truncated documentation |
+============================================================================================================+=========================+
| :py:meth:`_create_model <mlprodict.sklapi.onnx_tokenizer.GPT2TokenizerTransformer._create_model>`          |                         |
+------------------------------------------------------------------------------------------------------------+-------------------------+
| :py:meth:`_create_model <mlprodict.sklapi.onnx_tokenizer.SentencePieceTokenizerTransformer._create_model>` |                         |
+------------------------------------------------------------------------------------------------------------+-------------------------+




Methods
+++++++

+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| method                                                                                                   | truncated documentation                                                                            |
+==========================================================================================================+====================================================================================================+
| :py:meth:`__getstate__ <mlprodict.sklapi.onnx_tokenizer.GPT2TokenizerTransformer.__getstate__>`          |                                                                                                    |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :py:meth:`__getstate__ <mlprodict.sklapi.onnx_tokenizer.SentencePieceTokenizerTransformer.__getstate__>` |                                                                                                    |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :py:meth:`__getstate__ <mlprodict.sklapi.onnx_tokenizer.TokenizerTransformerBase.__getstate__>`          |                                                                                                    |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :py:meth:`__init__ <mlprodict.sklapi.onnx_tokenizer.GPT2TokenizerTransformer.__init__>`                  |                                                                                                    |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :py:meth:`__init__ <mlprodict.sklapi.onnx_tokenizer.SentencePieceTokenizerTransformer.__init__>`         |                                                                                                    |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :py:meth:`__init__ <mlprodict.sklapi.onnx_tokenizer.TokenizerTransformerBase.__init__>`                  |                                                                                                    |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :py:meth:`__setstate__ <mlprodict.sklapi.onnx_tokenizer.GPT2TokenizerTransformer.__setstate__>`          |                                                                                                    |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :py:meth:`__setstate__ <mlprodict.sklapi.onnx_tokenizer.SentencePieceTokenizerTransformer.__setstate__>` |                                                                                                    |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :py:meth:`__setstate__ <mlprodict.sklapi.onnx_tokenizer.TokenizerTransformerBase.__setstate__>`          |                                                                                                    |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :meth:`fit <mlprodict.sklapi.onnx_tokenizer.GPT2TokenizerTransformer.fit>`                               | The model is not trains this method is still needed to set the instance up and ready to transform. |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :meth:`fit <mlprodict.sklapi.onnx_tokenizer.SentencePieceTokenizerTransformer.fit>`                      | The model is not trains this method is still needed to set the instance up and ready to transform. |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :meth:`transform <mlprodict.sklapi.onnx_tokenizer.GPT2TokenizerTransformer.transform>`                   | Applies the tokenizers on an array of strings.                                                     |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| :meth:`transform <mlprodict.sklapi.onnx_tokenizer.SentencePieceTokenizerTransformer.transform>`          | Applies the tokenizers on an array of strings.                                                     |
+----------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+


Documentation
+++++++++++++

.. automodule:: mlprodict.sklapi.onnx_tokenizer
    :members:
    :special-members: __init__
    :show-inheritance: