Contrib Operator Schemas#

This file is automatically generated from the registered contrib operator schemas by this script. Do not modify directly.

com.microsoft#

com.microsoft.Attention#

Multi-Head Attention that can be either unidirectional (like GPT-2) or bidirectional (like BERT).

The weights for input projection of Q, K and V are merged. The data is stacked on the second dimension. Its shape is (input_hidden_size, hidden_size + hidden_size + v_hidden_size). Here hidden_size is the hidden dimension of Q and K, and v_hidden_size is that of V.

The mask_index is optional. Besides raw attention mask with shape (batch_size, total_sequence_length) or (batch_size, sequence_length, total_sequence_length) with value 0 for masked and 1 otherwise, we support other two formats: When input has right-side padding, mask_index is one dimension with shape (batch_size), where value is actual sequence length excluding padding. When input has left-side padding, mask_index has shape (2 * batch_size), where the values are the exclusive end positions followed by the inclusive start positions.

When unidirectional is 1, each token only attends to previous tokens.

Both past and present state are optional. They shall be used together, and not allowed to use only one of them.

When weights is not provided, key and value are required. In this situation, MatMul for input projection is excluded, and input is the query after projection. The bias is included for performance consideration.

The qkv_hidden_sizes is required only when K and V have different hidden sizes.

When there is past state, hidden dimension for Q, K and V shall be the same.

The total_sequence_length is past_sequence_length + kv_sequence_length. Here kv_sequence_length is the length of K or V. For self attention, kv_sequence_length equals to sequence_length (sequence length of Q). For cross attention, query and key might have different lengths.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

num_heads : int (required)
Number of attention heads
qkv_hidden_sizes : list of ints
Hidden dimension of Q, K, V: hidden_size, hidden_size and v_hidden_size
unidirectional : int
Whether every token can only attend to previous tokens. Default value is 0.

Inputs (3 - 8)#

input (optional) : T
Input tensor with shape (batch_size, sequence_length, input_hidden_size) when weights is available, or query tensor with shape (batch_size, sequence_length, hidden_size) when weights is not available.
weights (optional) : T
Merged Q/K/V weights with shape (input_hidden_size, hidden_size + hidden_size + v_hidden_size)
bias : T
Bias tensor with shape (hidden_size + hidden_size + v_hidden_size) for input projection
mask_index (optional) : M
Attention mask with shape (batch_size, 1, max_sequence_length, max_sequence_length), (batch_size, total_sequence_length) or (batch_size, sequence_length, total_sequence_length), or index with shape (batch_size) or (2 * batch_size).
past (optional) : T
past state for key and value with shape (2, batch_size, num_heads, past_sequence_length, head_size)
extra_add (optional) : T
additional add to QxK' with shape (batch_size, num_heads, sequence_length, total_sequence_length)
key (optional) : T
Input for key with shape (batch_size, kv_sequence_length, hidden_size). Required when weights is not available.
value (optional) : T
Input for key with shape (batch_size, kv_sequence_length, v_hidden_size). Required when weights is not available.

Outputs (1 - 2)#

output : T
3D output tensor with shape (batch_size, sequence_length, v_hidden_size)
present (optional) : T
past state for key and value with shape (2, batch_size, num_heads, total_sequence_length, head_size)

Type Constraints#

T : tensor(float), tensor(float16)
Constrain input and output types to float tensors.
M : tensor(int32)
Constrain mask index to integer types

com.microsoft.AttnLSTM#

Computes an one-layer RNN where its RNN Cell is an AttentionWrapper wrapped a LSTM Cell. The RNN layer contains following basic component: LSTM Cell, Bahdanau Attention Mechanism, AttentionWrapp.

Activation functions:

Relu(x)                - max(0, x)

Tanh(x)                - (1 - e^{-2x})/(1 + e^{-2x})

Sigmoid(x)             - 1/(1 + e^{-x})

(NOTE: Below are optional)

Affine(x)              - alpha*x + beta

LeakyRelu(x)           - x if x >= 0 else alpha * x

ThresholdedRelu(x)     - x if x >= alpha else 0

ScaledTanh(x)          - alpha*Tanh(beta*x)

HardSigmoid(x)         - min(max(alpha*x + beta, 0), 1)

Elu(x)                 - x if x >= 0 else alpha*(e^x - 1)

Softsign(x)            - x/(1 + |x|)

Softplus(x)            - log(1 + e^x)

Softmax(x)             - exp(x) / sum(exp(x))

Bahdanau Attention Mechanism: M - Memory tensor.

  `VALUES` - masked Memory by its real sequence length.

  `MW` - Memory layer weight.

  `KEYS` - Processed memory tensor by the memory layer.
           KEYS = M * MW

  `Query` - Query tensor, normally at specific time step in sequence.

  `QW` - Query layer weight in the attention mechanism

  `PQ` - processed query,  = `Query` * `QW`

  `V' - attention vector

  `ALIGN` - calculated alignment based on Query and KEYS
      ALIGN = softmax(reduce_sum(`V` * Tanh(`KEYS` + `PQ`)))

  `CONTEXT` - context based on `ALIGN` and `VALUES`
      CONTEXT = `ALIGN` * `VALUES`

LSTM Cell: X - input tensor concat with attention state in the attention wrapper

`i` - input gate

`o` - output gate

`f` - forget gate

`c` - cell gate

`t` - time step (t-1 means previous time step)

`W[iofc]` - W parameter weight matrix for input, output, forget, and cell gates

`R[iofc]` - R recurrence weight matrix for input, output, forget, and cell gates

`Wb[iofc]` - W bias vectors for input, output, forget, and cell gates

`Rb[iofc]` - R bias vectors for input, output, forget, and cell gates

`P[iof]`  - P peephole weight vector for input, output, and forget gates

`WB[iofc]` - W parameter weight matrix for backward input, output, forget, and cell gates

`RB[iofc]` - R recurrence weight matrix for backward input, output, forget, and cell gates

`WBb[iofc]` - W bias vectors for backward input, output, forget, and cell gates

`RBb[iofc]` - R bias vectors for backward input, output, forget, and cell gates

`PB[iof]`  - P peephole weight vector for backward input, output, and forget gates

`H` - Hidden state

`num_directions` - 2 if direction == bidirectional else 1

Equations (Default: f=Sigmoid, g=Tanh, h=Tanh):

  - it = f(Xt*(Wi^T) + Ht-1*(Ri^T) + Pi (.) Ct-1 + Wbi + Rbi)

  - ft = f(Xt*(Wf^T) + Ht-1*(Rf^T) + Pf (.) Ct-1 + Wbf + Rbf)

  - ct = g(Xt*(Wc^T) + Ht-1*(Rc^T) + Wbc + Rbc)

  - Ct = ft (.) Ct-1 + it (.) ct

  - ot = f(Xt*(Wo^T) + Ht-1*(Ro^T) + Po (.) Ct + Wbo + Rbo)

  - Ht = ot (.) h(Ct)

AttentionWrapp Notations: `lstm()’ - wrapped inner cell. Ht, Ct = lstm(concat(Xt, ATTNt-1), Ct-1)

`am()` - attention mechanism the wrapper used.
         CONTEXTt, ALIGNt = am(Ht, ALIGNt-1)

`AW` - attention layer weights, optional.

`ATTN` - attention state, initial is zero. If `AW` provided, it is the output of the attention layer,
              ATTNt = concat(Ht, CONTEXTt) * AW
         otherwise,
              ATTNt = CONTEXTt

RNN layer output: Y - if needed is the sequence of Ht from lstm cell.

`Y_h` - is the last valid H from lstm cell.

`Y_c` - is the last valid C from lstm cell.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

activation_alpha : list of floats
Optional scaling values used by some activation functions. The values are consumed in the order of activation functions, for example (f, g, h) in LSTM. Default values are the same as of corresponding ONNX operators.For example with LeakyRelu, the default alpha is 0.01.
activation_beta : list of floats
Optional scaling values used by some activation functions. The values are consumed in the order of activation functions, for example (f, g, h) in LSTM. Default values are the same as of corresponding ONNX operators.
activations : list of strings
A list of 3 (or 6 if bidirectional) activation functions for input, output, forget, cell, and hidden. The activation functions must be one of the activation functions specified above. Optional: See the equations for default if not specified.
clip : float
Cell clip threshold. Clipping bounds the elements of a tensor in the range of [-threshold, +threshold] and is applied to the input of activations. No clip if not specified.
direction : string
Specify if the RNN is forward, reverse, or bidirectional. Must be one of forward (default), reverse, or bidirectional.
hidden_size : int
Number of neurons in the hidden layer.
input_forget : int
Couple the input and forget gates if 1, default 0.

Inputs (3 - 14)#

X : T
The input sequences packed (and potentially padded) into one 3-D tensor with the shape of `[seq_length, batch_size, input_size]`
W : T
The weight tensor for the gates. Concatenation of `W[iofc]` and `WB[iofc]` (if bidirectional) along dimension 0. The tensor has shape `[num_directions, 4*hidden_size, input_size]`.
R : T
The recurrence weight tensor. Concatenation of `R[iofc]` and `RB[iofc]` (if bidirectional) along dimension 0. This tensor has shape `[num_directions, 4*hidden_size, hidden_size]`.
B (optional) : T
The bias tensor for input gate. Concatenation of `[Wb[iofc], Rb[iofc]]`, and `[WBb[iofc], RBb[iofc]]` (if bidirectional) along dimension 0. This tensor has shape `[num_directions, 8*hidden_size]`. Optional: If not specified - assumed to be 0.
sequence_lens (optional) : T1
Optional tensor specifying lengths of the sequences in a batch. If not specified - assumed all sequences in the batch to have length `seq_length`. It has shape `[batch_size]`
initial_h (optional) : T
Optional initial value of the hidden. If not specified - assumed to be 0. It has shape `[num_directions, batch_size, hidden_size]`.
initial_c (optional) : T
Optional initial value of the cell. If not specified - assumed to be 0. It has shape `[num_directions, batch_size, hidden_size]`.
P (optional) : T
The weight tensor for peepholes. Concatenation of `P[iof]` and `PB[iof]` (if bidirectional) along dimension 0. It has shape `[num_directions, 3*hidde_size]`. Optional: If not specified - assumed to be 0.
QW (optional) : T
The weight tensor of the query layer in the attention mechanism. Should be of shape `[num_directions, am_query_depth(hidden_size of lstm), am_attn_size]`
MW (optional) : T
The weight tensor of the memory layer in the attention mechanism. Should be of shape `[num_directions, memory_depth, am_attn_size]`
V (optional) : T
The attention_v tensor in the attention mechanism. Should be of shape `[num_directions, am_attn_size]`
M (optional) : T
The sequence of the memory (input) for attention mechanism. Should be of `[batch_size, max_memory_step, memory_depth]`
memory_seq_lens (optional) : T1
The sequence length of the input memory for the attention mechanism. Should be of `[batch_size]`
AW (optional) : T
The weights of attention layer in the attention wrapper. If exists, should be of shape `[num_directions, memory_depth+hidden_size, aw_attn_size]. Please note that attention mechanism context depth is also memory_depth in the attention mechanism.`

Outputs (0 - 3)#

Y (optional) : T
A tensor that concats all the intermediate output values of the hidden. It has shape `[seq_length, num_directions, batch_size, hidden_size]`
Y_h (optional) : T
The last output value of the hidden. It has shape `[num_directions, batch_size, hidden_size]`.
Y_c (optional) : T
The last output value of the cell. It has shape `[num_directions, batch_size, hidden_size]`.

Type Constraints#

T : tensor(float), tensor(double)
Constrain input and output types to float tensors.
T1 : tensor(int32)
Constrain seq_lens to integral tensors.

com.microsoft.BeamSearch#

Beam Search for text generation. Supports GPT-2 decoder.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

decoder : graph (required)
Decoder subgraph to execute in a loop.
decoder_start_token_id : int
The id of the token that indicates decoding starts.
early_stopping : int
early stop or not
encoder : graph
The subgraph for initialization of encoder and decoder. It will be called once before decoder subgraph.
eos_token_id : int (required)
The id of the end-of-sequence token
model_type : int
model type: 0 for GPT-2; 1 for encoder decoder like T5
no_repeat_ngram_size : int
no repeat ngrams size
pad_token_id : int (required)
The id of the padding token

Inputs (5 - 10)#

input_ids : I
The sequence used as a prompt for the generation. Shape is (batch_size, sequence_length)
max_length : I
The maximum length of the sequence to be generated. Shape is (1)
min_length (optional) : I
The minimum length below which the score of eos_token_id is set to -Inf. Shape is (1)
num_beams : I
Number of beams for beam search. 1 means no beam search. Shape is (1)
num_return_sequences : I
The number of returned sequences in the batch. Shape is (1)
length_penalty (optional) : T
Exponential penalty to the length. Default value 1.0 means no penalty.Value > 1.0 encourages longer sequences, while values < 1.0 produces shorter sequences.Shape is (1,)
repetition_penalty (optional) : T
The parameter for repetition penalty. Default value 1.0 means no penalty. Accepts value > 0.0. Shape is (1)
vocab_mask (optional) : M
Mask of vocabulary. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (vacab_size)
prefix_vocab_mask (optional) : M
Mask of vocabulary for first step. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (batch_size, vocab_size)
attention_mask (optional) : I
Custom attention mask. Shape is (batch_size, sequence_length)

Outputs (1 - 3)#

sequences : I
Word IDs of generated sequences. Shape is (batch_size, num_return_sequences, max_sequence_length)
sequences_scores (optional) : T
Final beam score of the generated sequences. Shape is (batch_size, num_return_sequences)
scores (optional) : T
Processed beam scores for each vocabulary token at each generation step.Beam scores consisting of log softmax scores for each vocabulary token and sum of log softmax of previously generated tokens in this beam.Shape is (max_length - sequence_length, batch_size, num_beams, vocab_size)

Type Constraints#

T : tensor(float)
Constrain input and output types to float tensors.
I : tensor(int32)
Constrain to integer types
M : tensor(int32)
Constrain mask to integer types

com.microsoft.BiasDropout#

output, dropout_mask = Dropout(data + bias, ratio) + residual, Intended to specialize the dropout pattern commonly found in transformer models.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

seed : int
(Optional) Seed to the random generator, if not specified we will auto generate one.

Inputs (2 - 5)#

data : T
The input data as Tensor.
bias : T
The bias input, a vector with the same shape as last dim of data OR same shape with data
residual (optional) : T
The residual input, must have the same shape as data
ratio (optional) : T1
The ratio of random dropout, with value in [0, 1). If this input was not set, or if it was set to 0, the output would be a simple copy of the input. If it's non-zero, output will be a random dropout of the scaled input, which is typically the case during training. It is an optional value, if not specified it will default to 0.5.
training_mode (optional) : T2
If set to true then it indicates dropout is being used for training. It is an optional value hence unless specified explicitly, it is false. If it is false, ratio is ignored and the operation mimics inference mode where nothing will be dropped from the input data and if mask is requested as output it will contain all ones.

Outputs (1 - 2)#

output : T
The output.
mask (optional) : T2
The output mask of dropout.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input and output types to float tensors.
T1 : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input 'ratio' types to float tensors.
T2 : tensor(bool)
Constrain output 'mask' types to boolean tensors.

com.microsoft.BiasGelu#

Bias Gelu. It’s an extension of Gelu. It takes the sum of input A and bias input B as the input of Gelu activation.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs#

A : T
The normal input data.
B : T
The bias input data that is a 1D tensor.

Outputs#

C : T
The output.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input and output types to float tensors.

com.microsoft.BiasSoftmax#

Y = softmax(scores + bias)) with simple broadcast on bias. Intended to specialize softmax(scores + additive_mask) commonly found in transformer models.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

axis : int
apply softmax to elements for dimensions axis or higher
is_inner_broadcast : int (required)
true if broadcast bias across input for dimensions broadcast_axis to axis-1, otherwise broadcast bias across input for dimensions 0 to broadcast_axis - 1

Inputs#

data : T
The input data as Tensor.
bias : T
The bias (or mask) as Tensor.

Outputs#

output : T
The output.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double)
Constrain input and output types to float tensors.

com.microsoft.BifurcationDetector#

Component for aggressive decoding. Find the bifurcation index of predicted tokens, between source tokens, starting from previous suffix match index, and predicted tokens. Concat predicted tokens, starting from bifurcation index, to the back of current tokens. This forms the output tokens. Detect suffix match index in source tokens, between source tokens and output tokens. Detection is based on finding the appearances of last n-gram in output tokens in source tokens. A match is considered found if source tokens contain a single matching n-gram. Return the index of the start of the n-gram in source tokens. No matching if found if src tokens contain multiple or zero matching n-grams. Return -1.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

max_ngram_size : int
The maximum NGram size for suffix matching.
min_ngram_size : int
The minimum NGram size for suffix matching.

Inputs (3 - 4)#

src_tokens : T
Encoder input ids.
cur_tokens : T
Decoder input ids.
prev_suffix_match_idx : T
Previous suffix match index
pred_tokens (optional) : T
Predicted token ids from aggressive decoding

Outputs#

tokens : T
Decoder input ids after merging predicted tokens
suffix_match_idx : T
new suffix match index

Type Constraints#

T : tensor(int64)
Constrain to integer types.

com.microsoft.BitmaskBiasDropout#

output, dropout_bitmask = Dropout(data + bias, ratio) + residual, Intended to specialize the dropout pattern commonly found in transformer models.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

seed : int
(Optional) Seed to the random generator, if not specified we will auto generate one.

Inputs (2 - 5)#

data : T
The input data as Tensor.
bias : T
The bias input, a vector with the same shape as last dim of data OR same shape with data
residual (optional) : T
The residual input, must have the same shape as data
ratio (optional) : T1
The ratio of random dropout, with value in [0, 1). If this input was not set, or if it was set to 0, the output would be a simple copy of the input. If it's non-zero, output will be a random dropout of the scaled input, which is typically the case during training. It is an optional value, if not specified it will default to 0.5.
training_mode (optional) : T2
If set to true then it indicates dropout is being used for training. It is an optional value hence unless specified explicitly, it is false. If it is false, ratio is ignored and the operation mimics inference mode where nothing will be dropped from the input data and if mask is requested as output it will contain all ones.

Outputs (1 - 2)#

output : T
The output.
mask (optional) : T3
The output mask of dropout.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input and output types to float tensors.
T1 : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input 'ratio' types to float tensors.
T2 : tensor(bool)
Constrain input 'training_mode' types to boolean tensors.
T3 : tensor(uint32)
Constrain output 'mask' types to uint32 tensors.

com.microsoft.BitmaskDropout#

BitmaskDropout takes an input floating-point tensor, an optional input ratio (floating-point scalar) and an optional input training_mode (boolean scalar). It produces two tensor outputs: output (floating-point tensor) and mask (optional Tensor<uint32>). If training_mode is true then the output Y will be a random dropout. Note that this Dropout scales the masked input data by the following equation, so to convert the trained model into inference mode, the user can simply not pass training_mode input or set it to false.

output = scale * data * mask,

where

scale = 1. / (1. - ratio).

This op functions in much the same was as Dropout-11 and Dropout-13 do, execpt that the mask is output as a bit-packed uint32 tensor, instead of a boolean tensor.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

seed : int
(Optional) Seed to the random generator, if not specified we will auto generate one.

Inputs (1 - 3)#

data : T
The input data as Tensor.
ratio (optional) : T1
The ratio of random dropout, with value in [0, 1). If this input was not set, or if it was set to 0, the output would be a simple copy of the input. If it's non-zero, output will be a random dropout of the scaled input, which is typically the case during training. It is an optional value, if not specified it will default to 0.5.
training_mode (optional) : T2
If set to true then it indicates dropout is being used for training. It is an optional value hence unless specified explicitly, it is false. If it is false, ratio is ignored and the operation mimics inference mode where nothing will be dropped from the input data and if mask is requested as output it will contain all ones.

Outputs (1 - 2)#

output : T
The output.
mask (optional) : T3
The bit-packed output mask.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input and output types to float tensors.
T1 : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input 'ratio' types to float tensors.
T2 : tensor(bool)
Constrain 'training_mode' to boolean tensor.
T3 : tensor(uint32)
Constrain output 'mask' types to bit-packed uint32 tensor.

com.microsoft.CDist#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

metric : string
The distance metric to use. If a string, the distance function can be "braycurtis", "canberra", "chebyshev", "cityblock", "correlation", "cosine", "dice", "euclidean", "hamming", "jaccard", "jensenshannon", "kulsinski", "mahalanobis", "matching", "minkowski", "rogerstanimoto", "russellrao", "seuclidean", "sokalmichener", "sokalsneath", "sqeuclidean", "wminkowski", "yule".

Inputs#

A : T
2D matrix with shape (M,N)
B : T
2D matrix with shape (K,N)

Outputs#

C : T
A 2D Matrix that represents the distance between each pair of the two collections of inputs.

Type Constraints#

T : tensor(float), tensor(double)
Constrains input to only numeric types.

com.microsoft.ComplexMul#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs#

A : T
input_0
B : T
input_1

Outputs#

C : T
output tensor

Type Constraints#

T : tensor(float), tensor(double), tensor(float16)
Constrain input and output types to float or half tensors.

com.microsoft.ComplexMulConj#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs#

A : T
input_0
B : T
input_1

Outputs#

C : T
output tensor

Type Constraints#

T : tensor(float), tensor(double), tensor(float16)
Constrain input and output types to float or half tensors.

com.microsoft.ConvTransposeWithDynamicPads#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

auto_pad : string
dilations : list of ints
group : int
kernel_shape : list of ints
output_padding : list of ints
strides : list of ints

Inputs (2 - 4)#

X : T
W : T
Pads (optional) : tensor(int64)
B (optional) : T

Outputs#

Y : T

Type Constraints#

T : tensor(float16), tensor(float), tensor(double)
Constrain input and output types to float tensors

com.microsoft.CropAndResize#

Extracts crops from the input image tensor and resizes them using bilinear sampling or nearest neighbor sampling (possibly with aspect ratio change) to a common output size specified by crop_height and crop_width. Returns a tensor with crops from the input image at positions defined at the bounding box locations in boxes. The cropped boxes are all resized (with bilinear or nearest neighbor interpolation) to a fixed size = [crop_height, crop_width]. The result is a 4-D tensor [num_boxes, crop_height, crop_width, depth]. The resizing is corner aligned.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

extrapolation_value : float
Value used for extrapolation, when applicable. Default is 0.0f.
mode : string
The pooling method. Two modes are supported: 'bilinear' and 'nearest'. Default is 'bilinear'.

Inputs#

X : T1
Input data tensor from the previous operator; 4-D feature map of shape (N, C, H, W), where N is the batch size, C is the number of channels, and H and W are the height and the width of the data.
rois : T1
RoIs (Regions of Interest) to pool over; rois is 2-D input of shape (num_rois, 4) given as [[y1, x1, y2, x2], ...]. The RoIs' coordinates are normalized in the coordinate system of the input image. Each coordinate set has a 1:1 correspondence with the 'batch_indices' input.
batch_indices : T2
1-D tensor of shape (num_rois,) with each element denoting the index of the corresponding image in the batch.
crop_size : T2
1-D tensor of 2 elements: [crop_height, crop_width]. All cropped image patches are resized to this size. Both crop_height and crop_width need to be positive.

Outputs#

Y : T1
RoI pooled output, 4-D tensor of shape (num_rois, C, crop_height, crop_width). The r-th batch element Y[r-1] is a pooled feature map corresponding to the r-th RoI X[r-1].

Type Constraints#

T1 : tensor(float16), tensor(float), tensor(double)
Constrain types to float tensors.
T2 : tensor(int32)
Constrain types to int tensors.

com.microsoft.DecoderAttention#

This DecoderAttention supports self attention and cross attention, key and value cache, and key_padding_mask. The attention mask is not support at the moment. Some boolean parameters are passed by runtime input for generic purpose

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

num_heads : int (required)
Number of attention heads

Inputs#

query : T
3D input tensor with shape (sequence_length, batch_size, hidden_size), hidden_size = num_heads * head_size
key : T
3D input tensor with shape (total_sequence_length, batch_size, hidden_size)
q_weight : T
2D input tensor with shape (hidden_size, hidden_size)
kv_weight : T
2D input tensor with shape (hidden_size, 2 * hidden_size)
bias : T
1D input tensor with shape (3 * hidden_size)
key_padding_mask (optional) : B
2D input tensor with shape (batch_size, total_sequence_length)
key_cache (optional) : T
input tensor with shape (batch_size, num_heads, sequence_length or total_sequence_length, head_size)
value_cache (optional) : T
input tensor with shape (batch_size, num_heads, sequence_length or total_sequence_length, head_size)
static_kv : B
If static_kv = true, cross-attention; else self-attention
use_past : B
If use_past = true, use cache; else no cache
has_layer_state : B
If has_layer_state = true, layer_state = {} or [a,b]; else layer_state = None
has_key_padding_mask : B
has_key_padding_mask or not

Outputs (1 - 3)#

output : T
3D output tensor with shape (sequence_length, batch_size, hidden_size)
new_key_cache (optional) : T
output tensor with shape (batch_size, num_heads, new sequence_length, head_size)
new_value_cache (optional) : T
output tensor with shape (batch_size, num_heads, new sequence_length, head_size)

Type Constraints#

T : tensor(float), tensor(float16)
Constrain input and output types to float and float16 tensors.
B : tensor(bool)
Constrain key_padding_mask to bool tensors.

com.microsoft.DequantizeBFP#

The BFP dequantization operator. It consumes the raw BFP data and some metadata such as the shape and strides of the original tensor and computes the dequantized tensor. More documentation on the BFP format can be found in this paper: https://www.microsoft.com/en-us/research/publication/pushing-the-limits-of-narrow-precision-inferencing-at-cloud-scale-with-microsoft-floating-point/

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

bfp_type : int (required)
The type of BFP - must match with the BFPType enum
block_dim : int
Each bounding box spans this dimension.Typically, the block dimension corresponds to the reduction dimension of the matrix multipication that consumes the output of this operator.For example, for a 2D matrix multiplication A@W, QuantizeBFP(A) would use block_dim 1 and QuantizeBFP(W) would use block_dim 0.The default is the last dimension.
dtype : int
The datatype to dequantize to.

Inputs#

x : T1
1-D, contiguous, raw, BFP data to be de-quantized.
shape : T2
shape of the original tensor.
strides : T2
strides of the original tensor.

Outputs#

y : T3
de-quantized tensor.

Type Constraints#

T1 : tensor(uint8)
Constrain the input to uint8.
T2 : tensor(int64)
Constrain shape and strides to uint64.
T3 : tensor(float), tensor(float16), tensor(bfloat16)
Constrain y to float and bfloat16.

com.microsoft.DequantizeLinear#

The linear dequantization operator. It consumes a quantized data, a scale, a zero point and computes the full precision data. The dequantization formula is y = (x - x_zero_point) * x_scale. Scale and zero point must have same shape. They must be either scalar (per tensor) or 1-D tensor (per ‘axis’).

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

axis : int
The axis along which same quantization parameters are applied. It's optional.If it's not specified, it means per-tensor quantization and input 'x_scale' and 'x_zero_point' must be scalars.If it's specified, it means per 'axis' quantization and input 'x_scale' and 'x_zero_point' must be 1-D tensors.

Inputs#

x : T1
N-D quantized Input tensor to be de-quantized.
x_scale : T2
Scale for input 'x'. It could be a scalar or a 1-D tensor, which means a per-tensor or per-axis quantization.If it's a 1-D tensor, its number of elements should be equal to the dimension value of 'axis' dimension of input 'x'.
x_zero_point : T1
Zero point for input 'x'. It could be a scalar or a 1-D tensor, which means a per-tensor or per-axis quantization.If it's a 1-D tensor, its number of elements should be equal to the dimension value of 'axis' dimension of input 'x'.

Outputs#

y : T2
N-D full precision output tensor. It has same shape as input 'x'.

Type Constraints#

T1 : tensor(int8), tensor(uint8)
Constrain 'x' and 'x_zero_point' to 8-bit integer tensors.
T2 : tensor(float16), tensor(float)
Constrain 'y', 'x_scale' to float tensors.

com.microsoft.DequantizeWithOrder#

Dequantize input matrix to specific layout used in cublaslt. attr to specify output type, float16 or float32

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

order_input : int (required)
cublasLt order of input matrix. See the schema of QuantizeWithOrder for order definition.
order_output : int (required)
cublasLt order of output matrix
to : int (required)
The output data type, only support TensorProto_DataType_FLOAT (1) and TensorProto_DataType_FLOAT16 (10)

Inputs#

input : Q
TODO: input tensor of (ROWS, COLS). if less than 2d, will broadcast to (1, X). If 3d, it is treated as (B, ROWS, COS)
scale_input : S
scale of the input

Outputs#

output : F
output tensor

Type Constraints#

Q : tensor(int8)
Constrain input and output types to int8 tensors.
F : tensor(float16), tensor(float)
Constrain to float types
S : tensor(float)
Constrain Scale to float32 types

com.microsoft.DynamicQuantizeLSTM#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

activation_alpha : list of floats
Optional scaling values used by some activation functions. The values are consumed in the order of activation functions, for example (f, g, h) in LSTM. Default values are the same as of corresponding ONNX operators.For example with LeakyRelu, the default alpha is 0.01.
activation_beta : list of floats
Optional scaling values used by some activation functions. The values are consumed in the order of activation functions, for example (f, g, h) in LSTM. Default values are the same as of corresponding ONNX operators.
activations : list of strings
A list of 3 (or 6 if bidirectional) activation functions for input, output, forget, cell, and hidden. The activation functions must be one of the activation functions specified above. Optional: See the equations for default if not specified.
clip : float
Cell clip threshold. Clipping bounds the elements of a tensor in the range of [-threshold, +threshold] and is applied to the input of activations. No clip if not specified.
direction : string
Specify if the RNN is forward, reverse, or bidirectional. Must be one of forward (default), reverse, or bidirectional.
hidden_size : int
Number of neurons in the hidden layer
input_forget : int
Couple the input and forget gates if 1.

Inputs#

X : T
The input sequences packed (and potentially padded) into one 3-D tensor with the shape of `[seq_length, batch_size, input_size]`.
W : T2
The weight tensor for the gates. Concatenation of `W[iofc]` and `WB[iofc]` (if bidirectional) along dimension 0. The tensor has shape `[num_directions, input_size, 4*hidden_size]`.
R : T2
The recurrence weight tensor. Concatenation of `R[iofc]` and `RB[iofc]` (if bidirectional) along dimension 0. This tensor has shape `[num_directions, hidden_size, 4*hidden_size]`.
B (optional) : T
The bias tensor for input gate. Concatenation of `[Wb[iofc], Rb[iofc]]`, and `[WBb[iofc], RBb[iofc]]` (if bidirectional) along dimension 0. This tensor has shape `[num_directions, 8*hidden_size]`. Optional: If not specified - assumed to be 0.
sequence_lens (optional) : T1
Optional tensor specifying lengths of the sequences in a batch. If not specified - assumed all sequences in the batch to have length `seq_length`. It has shape `[batch_size]`.
initial_h (optional) : T
Optional initial value of the hidden. If not specified - assumed to be 0. It has shape `[num_directions, batch_size, hidden_size]`.
initial_c (optional) : T
Optional initial value of the cell. If not specified - assumed to be 0. It has shape `[num_directions, batch_size, hidden_size]`.
P (optional) : T
The weight tensor for peepholes. Concatenation of `P[iof]` and `PB[iof]` (if bidirectional) along dimension 0. It has shape `[num_directions, 3*hidde_size]`. Optional: If not specified - assumed to be 0.
W_scale : T
W's scale. Its size is [num_directions] for per-tensor/layer quantization, or [num_directions, 4*hidden_size] for per-channel quantization on the axis input_size.
W_zero_point : T2
W's zero point. Its size is [num_directions] for per-tensor/layer quantization, or [num_directions, 4*hidden_size] for per-channel quantization on the axis input_size.
R_scale : T
R's scale. Its size is [num_directions] for per-tensor/layer quantization, or [num_directions, 4*hidden_size] for per-channel quantization on the axis input_size.
R_zero_point : T2
R's zero point. Its size is [num_directions] for per-tensor/layer quantization, or [num_directions, 4*hidden_size] for per-channel quantization on the axis input_size.

Outputs (0 - 3)#

Y (optional) : T
A tensor that concats all the intermediate output values of the hidden. It has shape `[seq_length, num_directions, batch_size, hidden_size]`.
Y_h (optional) : T
The last output value of the hidden. It has shape `[num_directions, batch_size, hidden_size]`.
Y_c (optional) : T
The last output value of the cell. It has shape `[num_directions, batch_size, hidden_size]`.

Type Constraints#

T : tensor(float)
Constrain input and output types to float tensors.
T1 : tensor(int32)
Constrain seq_lens to integer tensor.
T2 : tensor(uint8), tensor(int8)
Constrain weights types to 8 bit tensors.

com.microsoft.DynamicQuantizeMatMul#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs (3 - 5)#

A : T1
N-dimensional matrix A
B : T2
N-dimensional matrix B
b_scale : T1
Scale of quantized input 'B'. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
b_zero_point (optional) : T2
Zero point tensor for input 'B'. It's optional and default value is 0. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
bias (optional) : T1
1D input tensor, whose dimension is same as B's last dimension

Outputs#

Y : T1
Matrix multiply results from A * B

Type Constraints#

T1 : tensor(float)
Constrain input A, b_scale and output Y data type as float tensor.
T2 : tensor(int8), tensor(uint8)
Constrain input B data type to 8-bit integer tensor.

com.microsoft.EmbedLayerNormalization#

EmbedLayerNormalization is the fusion of embedding layer in BERT model, with optional mask processing. The embedding layer takes input_ids (word IDs) and segment_ids (sentence IDs) to look up word_embedding, position_embedding, and segment_emedding; the embeddings are added then applied layer normalization using gamma and beta tensors. The last input mask is optional. If mask is provided, mask index (that is position of first 0 in mask, or number of words) will be calculated.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

epsilon : float
The epsilon value to use to avoid division by zero.

Inputs (7 - 9)#

input_ids : T1
2D words IDs with shape (batch_size, sequence_length)
segment_ids (optional) : T1
2D segment IDs with shape (batch_size, sequence_length)
word_embedding : T
2D with shape (,hidden_size)
position_embedding : T
2D with shape (, hidden_size)
segment_embedding (optional) : T
2D with shape (, hidden_size)
gamma : T
1D gamma tensor for layer normalization with shape (hidden_size)
beta : T
1D beta tensor for layer normalization with shape (hidden_size)
mask (optional) : T1
2D attention mask with shape (batch_size, sequence_length)
position_ids (optional) : T1
2D position ids with shape (batch_size, sequence_length) or (1, sequence_length)

Outputs (2 - 3)#

output : T
3D output tensor with shape (batch_size, sequence_length, hidden_size)
mask_index : T1
1D mask_index tensor with shape (batch_size)
embedding_sum (optional) : T
sum of word_embedding and position_embedding without layer normalization

Type Constraints#

T1 : tensor(int32)
Constrain input and output integer tensors types
T : tensor(float), tensor(float16)
Constrain input and output float tensors types.

com.microsoft.ExpandDims#

ExpandDims echo operator.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs#

X : T
input
axis : tensor(int32)
Specified axis to insert a dimension

Outputs#

Y : T
output

Type Constraints#

T : tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(float16), tensor(float), tensor(double), tensor(string), tensor(bool), tensor(complex64), tensor(complex128)
Constrain to any tensor type. If the dtype attribute is not provided this must be a valid output type.

com.microsoft.FastGelu#

GELU (Gaussian Error Linear Unit) approximation: Y=0.5X(1+tanh(0.797885X+0.035677XXX)) with an optional input of bias that will be added to X before GELU.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs (1 - 2)#

X : T
input tensor
bias (optional) : T
bias tensor

Outputs#

Y : T
output tensor

Type Constraints#

T : tensor(float), tensor(float16), tensor(bfloat16)
Constrain input and output types to float or half tensors.

com.microsoft.FusedConv#

The fused convolution operator schema is the same as Conv besides it includes an attribute activation.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

activation : string
activation_params : list of floats
auto_pad : string
dilations : list of ints
group : int
kernel_shape : list of ints
pads : list of ints
strides : list of ints

Inputs (2 - 4)#

X : T
W : T
B (optional) : T
Z (optional) : T

Outputs#

Y : T

Type Constraints#

T : tensor(float16), tensor(float), tensor(double)
Constrain input and output types to float tensors

com.microsoft.FusedGemm#

The FusedGemm operator schema is the same as Gemm besides it includes attributes activation and leaky_relu_alpha.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

activation : string
activation_alpha : float
activation_beta : float
activation_gamma : float
alpha : float
Scalar multiplier for the product of input tensors A * B.
beta : float
Scalar multiplier for input tensor C.
transA : int
Whether A should be transposed
transB : int
Whether B should be transposed

Inputs (2 - 3)#

A : T
Input tensor A. The shape of A should be (M, K) if transA is 0, or (K, M) if transA is non-zero.
B : T
Input tensor B. The shape of B should be (K, N) if transB is 0, or (N, K) if transB is non-zero.
C (optional) : T
Input tensor C. The shape of C should be unidirectional broadcastable to (M, N).

Outputs#

Y : T
Output tensor of shape (M, N).

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(uint32), tensor(uint64), tensor(int32), tensor(int64)
Constrain input and output types to float/int tensors.

com.microsoft.FusedMatMul#

Matrix product that behaves like numpy.matmul: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.matmul.html

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

alpha : float
Scalar multiplier for the product of the input tensors.
transA : int
Whether A should be transposed on the last two dimensions before doing multiplication
transB : int
Whether B should be transposed on the last two dimensions before doing multiplication
transBatchA : int
Whether A should be transposed on the 1st dimension and batch dimensions (dim-1 to dim-rank-2) before doing multiplication
transBatchB : int
Whether B should be transposed on the 1st dimension and batch dimensions (dim-1 to dim-rank-2) before doing multiplication

Inputs#

A : T
N-dimensional matrix A
B : T
N-dimensional matrix B

Outputs#

Y : T
Matrix multiply results

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input and output types to float tensors.

com.microsoft.GatherND#

Given data tensor of rank r >= 1, and indices tensor of rank q >= 1, gather slices of data into an output tensor of rank q - 1 + r - indices[-1]. Example 1: data = [[0,1],[2,3]] indices = [[0,0],[1,1]] output = [0,3] Example 2: data = [[0,1],[2,3]] indices = [[1],[0]] output = [[2,3],[0,1]] Example 3: data = [[[0,1],[2,3]],[[4,5],[6,7]]] indices = [[0,1],[1,0]] output = [[2,3],[4,5]] Example 4: data = [[[0,1],[2,3]],[[4,5],[6,7]]] indices = [[[0,1]],[[1,0]]] output = [[[2,3]],[[4,5]]]

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs#

data : T
Tensor of rank r >= 1.
indices : Tind
Tensor of rank q >= 1.

Outputs#

output : T
Tensor of rank q-1+r-indices[-1].

Type Constraints#

T : tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(float16), tensor(float), tensor(double), tensor(string), tensor(bool), tensor(complex64), tensor(complex128)
Constrain input and output types to any tensor type.
Tind : tensor(int32), tensor(int64)
Constrain indice type to int32 or int64

com.microsoft.Gelu#

Gaussian Error Linear Unit. A high-performing neural network activation function.The GELU nonlinearity is the expected transformation of a stochastic regularizer which randomly applies the identity or zero map to a neuron’s input. The GELU nonlinearity weights inputs by their magnitude, rather than gates inputs by their sign as in ReLUs.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs#

X : T
The input data as Tensor.

Outputs#

Y : T
The output.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input and output types to float tensors.

com.microsoft.GemmFastGelu#

It’s a fusion of MatMul and FastGelu.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs (2 - 3)#

X : T
input tensor
W : T
input tensor
bias (optional) : T
bias tensor

Outputs#

Y : T
output tensor

Type Constraints#

T : tensor(float), tensor(float16), tensor(bfloat16)
Constrain input and output types to float or half tensors.

com.microsoft.GreedySearch#

Greedy Search for text generation.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

decoder : graph (required)
Decoder subgraph to execute in a loop.
decoder_start_token_id : int
The id of the token that indicates decoding starts.
encoder : graph
The subgraph for initialization of encoder and decoder. It will be called once before decoder subgraph.
eos_token_id : int (required)
The id of the end-of-sequence token
model_type : int
model type: 0 for decoder only like GPT-2; 1 for encoder decoder like Bart
no_repeat_ngram_size : int
no repeat ngrams size
pad_token_id : int (required)
The id of the padding token

Inputs (2 - 7)#

input_ids : I
The sequence used as a prompt for the generation. Shape is (batch_size, sequence_length)
max_length : I
The maximum length of the sequence to be generated. Shape is (1)
min_length (optional) : I
The minimum length below which the score of eos_token_id is set to -Inf. Shape is (1)
repetition_penalty (optional) : T
The parameter for repetition penalty. Default value 1.0 means no penalty. Accepts value > 0.0. Shape is (1)
vocab_mask (optional) : I
Mask of vocabulary. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (vacab_size)
prefix_vocab_mask (optional) : I
Mask of vocabulary for first step. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (batch_size, vocab_size)
attention_mask (optional) : I
Custom attention mask. Shape is (batch_size, sequence_length)

Outputs#

sequences : I
Word IDs of generated sequences. Shape is (batch_size, max_sequence_length)

Type Constraints#

T : tensor(float)
Constrain input and output types to float tensors.
I : tensor(int32)
Constrain to integer types

com.microsoft.GridSample#

Given an input and a flow-field grid, computes the output using input values and pixel locations from grid. Currently, only spatial (4-D) inputs are supported. For input with shape (N, C, H, W) and grid with shape (N, H_out, W_out, 2), the output will have shape (N, C, H_out, W_out). For each output location output[n, :, h, w], the size-2 vector grid[n, h, w] specifies input pixel locations x and y, which are used to interpolate the output value output[n, :, h, w]. The GridSample operator is often used in doing grid generator and sampler in the Spatial Transformer Networks. See also in torch.nn.functional.grid_sample.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

align_corners : int
If align_corners=1, the extrema (-1 and 1) are considered as referring to the center points of the input's corner pixels. If align_corners=0, they are instead considered as referring to the corner points of the input's corner pixels, making the sampling more resolution agnostic.
mode : string
Three interpolation modes: bilinear (default), nearest and bicubic.
padding_mode : string
Support padding modes for outside grid values: `zeros`(default), `border`, `reflection`. zeros: use 0 for out-of-bound grid locations, border: use border values for out-of-bound grid locations, reflection: use values at locations reflected by the border for out-of-bound grid locations.

Inputs#

X : T1
4-D tensor of shape (N, C, H, W), where N is the batch size, C is the numbers of channels, H and W are the height and width of the input data.
Grid : T1
Input offset, 4-D tensor of shape (N, H_out, W_out, 2), where H_out and W_out are the height and width of grid and output, Grid specifies the sampling pixel locations normalized by the input spatial dimensions. Therefore, it should have most values in the range of [-1, 1]. If grid has values outside the range of [-1, 1], the corresponding outputs will be handled as defined by padding_mode.

Outputs#

Y : T2
4-D tensor of shape (N, C, H_out, W_out).

Type Constraints#

T1 : tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(float16), tensor(float), tensor(double), tensor(string), tensor(bool), tensor(complex64), tensor(complex128)
Constrain input types to all tensor types.
T2 : tensor(float16), tensor(float), tensor(double)
Constrain output types to float tensors.

com.microsoft.Inverse#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs#

X : T
Input tensor. Every matrix in the batch must be invertible.

Outputs#

Y : T
Output tensor of the same type and shape as the input tensor.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double)
Constrain input and output types to float tensors.

com.microsoft.Irfft#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

normalized : int
onesided : int
signal_ndim : int (required)

Inputs#

X : T
input tensor

Outputs#

Y : T
output tensor

Type Constraints#

T : tensor(float), tensor(double), tensor(float16)
Constrain input and output types to float or half tensors.

com.microsoft.LongformerAttention#

Longformer Self Attention with a local context and a global context. Tokens attend locally: Each token attends to its W previous tokens and W succeeding tokens with W being the window length. A selected few tokens attend globally to all other tokens.

The attention mask is of shape (batch_size, sequence_length), where sequence_length is a multiple of 2W after padding. Mask value < 0 (like -10000.0) means the token is masked, 0 otherwise.

Global attention flags have value 1 for the tokens attend globally and 0 otherwise.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

num_heads : int (required)
Number of attention heads
window : int (required)
One sided attention windows length W, or half of total window length

Inputs#

input : T
3D input tensor with shape (batch_size, sequence_length, hidden_size), hidden_size = num_heads * head_size
weight : T
2D input tensor with shape (hidden_size, 3 * hidden_size)
bias : T
1D input tensor with shape (3 * hidden_size)
mask : T
Attention mask with shape (batch_size, sequence_length)
global_weight : T
2D input tensor with shape (hidden_size, 3 * hidden_size)
global_bias : T
1D input tensor with shape (3 * hidden_size)
global : G
Global attention flags with shape (batch_size, sequence_length)

Outputs#

output : T
3D output tensor with shape (batch_size, sequence_length, hidden_size)

Type Constraints#

T : tensor(float), tensor(float16)
Constrain input and output types to float tensors.
G : tensor(int32)
Constrain to integer types

com.microsoft.MatMulInteger16#

Matrix product that behaves like numpy.matmul: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.matmul.html. The production MUST never overflow. The accumulation may overflow if and only if in 32 bits.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs#

A : T1
N-dimensional matrix A
B : T2
N-dimensional matrix B

Outputs#

Y : T3
Matrix multiply results from A * B

Type Constraints#

T1 : tensor(int16), tensor(uint16)
Constrain input A data types as 16-bit integer tensor
T2 : tensor(int16), tensor(uint16)
Constrain input B data types as 16-bit integer tensor
T3 : tensor(int32), tensor(uint32)
Constrain output Y data types as 32-bit integer tensor.T3 must be tensor(uint32) when both T1 and T2 are tensor(uint16),or must be tensor(int32) when either T1 or T2 is tensor(int16).

com.microsoft.MatMulIntegerToFloat#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs (4 - 7)#

A : T1
N-dimensional matrix A
B : T2
N-dimensional matrix B
a_scale : T3
Scale of quantized input 'A'. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'A'.
b_scale : T3
Scale of quantized input 'B'. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
a_zero_point (optional) : T1
Zero point tensor for input 'A'. It's optional and default value is 0. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'A'.
b_zero_point (optional) : T2
Zero point tensor for input 'B'. It's optional and default value is 0. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
bias (optional) : T3
1D input tensor, whose dimension is same as B's last dimension

Outputs#

Y : T3
Matrix multiply results from A * B

Type Constraints#

T1 : tensor(int8), tensor(uint8)
Constrain input A data type to 8-bit integer tensor.
T2 : tensor(int8), tensor(uint8)
Constrain input B data type to 8-bit integer tensor.
T3 : tensor(float)
Constrain input a_scale, b_scale and output Y data type as float tensor.

com.microsoft.MaxpoolWithMask#

For internal use.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

auto_pad : string
kernel_shape : list of ints
pads : list of ints
storage_order : int
strides : list of ints

Inputs#

X : T
M : tensor(int32)
mask

Outputs#

Y : T

Type Constraints#

T : tensor(float)
Constrain input0 and output types to float tensors

com.microsoft.MulInteger#

Performs element-wise binary quantized multiplication (with Numpy-style broadcasting support). “This operator supports multidirectional (i.e., Numpy-style) broadcasting” The output of this op is the int32 accumulated result of the mul operation

C (int32) = (A - A_zero_point) * (B - B_zero_point)

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs (3 - 4)#

A : T
First operand.
A_zero_point (optional) : T
Input A zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
B : T
Second operand.
B_zero_point (optional) : T
Input B zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.

Outputs#

C : T1
Constrain output to 32 bit tensor

Type Constraints#

T : tensor(uint8), tensor(int8)
Constrain input types to 8 bit signed and unsigned tensors.
T1 : tensor(int32)
Constrain output types to 32 bit tensors.

com.microsoft.MurmurHash3#

The underlying implementation is MurmurHash3_x86_32 generating low latency 32bits hash suitable for implementing lookup tables, Bloom filters, count min sketch or feature hashing.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

positive : int
If value is 1, output type is uint32_t, else int32_t. Default value is 1.
seed : int
Seed for the hashing algorithm, unsigned 32-bit integer, default to 0.

Inputs#

X : T1
An input tensor to hash.

Outputs#

Y : T2
32-bit hash value.

Type Constraints#

T1 : tensor(uint32), tensor(int32), tensor(uint64), tensor(int64), tensor(float), tensor(double), tensor(string)
Constrain input type to unsigned or signed 32-bit integer tensor, or string tensor. It should be utf-8 encoded if using unicode.
T2 : tensor(uint32), tensor(int32)
Constrain output type to unsigned and signed 32-bit integer tensor.

com.microsoft.NGramRepeatBlock#

Enforce no repetition of n-grams. Scores are set to -inf for tokens that form a repeated n-gram if added to the back of the input_ids.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

ngram_size : int (required)
The NGram size.

Inputs#

input_ids : Tid
2D input tensor with shape (batch_size, sequence_length)
scores : T
2D input tensor with shape (batch_size, vocab_size)

Outputs#

scores_out : T
2D output tensor with shape (batch_size, vocab_size)

Type Constraints#

Tid : tensor(int64)
Constrain indices to integer types
T : tensor(float)
Constrain scores input and output types to float tensors.

com.microsoft.NhwcConv#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

auto_pad : string
dilations : list of ints
dilation value along each spatial axis of the filter. If not present, the dilation defaults is 1 along each spatial axis.
group : int
number of groups input channels and output channels are divided into.
kernel_shape : list of ints
The shape of the convolution kernel. If not present, should be inferred from input W.
pads : list of ints
strides : list of ints
Stride along each spatial axis. If not present, the stride defaults is 1 along each spatial axis.

Inputs (2 - 3)#

X : T
Input data tensor from previous layer; has size (N x C x H x W), where N is the batch size, C is the number of channels, and H and W are the height and width. Note that this is for the 2D image. Otherwise the size is (N x C x D1 x D2 ... x Dn). Optionally, if dimension denotation is in effect, the operation expects input data tensor to arrive with the dimension denotation of [DATA_BATCH, DATA_CHANNEL, DATA_FEATURE, DATA_FEATURE ...].
W : T
The weight tensor that will be used in the convolutions; has size (M x C/group x kH x kW), where C is the number of channels, and kH and kW are the height and width of the kernel, and M is the number of feature maps. For more than 2 dimensions, the kernel shape will be (M x C/group x k1 x k2 x ... x kn), where (k1 x k2 x ... kn) is the dimension of the kernel. Optionally, if dimension denotation is in effect, the operation expects the weight tensor to arrive with the dimension denotation of [FILTER_OUT_CHANNEL, FILTER_IN_CHANNEL, FILTER_SPATIAL, FILTER_SPATIAL ...]. Assuming zero based indices for the shape array, X.shape[1] == (W.shape[1] * group) == C and W.shape[0] mod G == 0. Or in other words FILTER_IN_CHANNEL multiplied by the number of groups should be equal to DATA_CHANNEL and the number of feature maps M should be a multiple of the number of groups G.
B (optional) : T
Optional 1D bias to be added to the convolution, has size of M.

Outputs#

Y : T
Output data tensor that contains the result of the convolution. The output dimensions are functions of the kernel size, stride size, and pad lengths.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double)
Constrain input and output types to float tensors.

com.microsoft.NhwcMaxPool#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

auto_pad : string
ceil_mode : int
dilations : list of ints
kernel_shape : list of ints (required)
pads : list of ints
strides : list of ints

Inputs#

x : T

Outputs#

y : T

Type Constraints#

T : tensor(int8), tensor(uint8)

com.microsoft.Pad#

Given data tensor, pads, mode, and value. Example: Insert 0 pads to the beginning of the second dimension. data = [ [1.0, 1.2], [2.3, 3.4], [4.5, 5.7], ] pads = [0, 2, 0, 0] output = [ [ [0.0, 0.0, 1.0, 1.2], [0.0, 0.0, 2.3, 3.4], [0.0, 0.0, 4.5, 5.7], ], ]

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

mode : string
Three modes: `constant`(default) - pads with a given constant value, `reflect` - pads with the reflection of the vector mirrored on the first and last values of the vector along each axis, `edge` - pads with the edge values of array

Inputs (2 - 3)#

data : T
Input tensor.
pads : tensor(int64)
Tensor of integers indicating the number of padding elements to add or remove (if negative) at the beginning and end of each axis. For 2D input tensor, it is the number of pixels. `pads` should be a 1D tensor of shape [2 * input_rank] or a 2D tensor of shape [1, 2 * input_rank]. `pads` format (1D example) should be as follow [x1_begin, x2_begin,...,x1_end, x2_end,...], where xi_begin is the number of pixels added at the beginning of axis `i` and xi_end, the number of pixels added at the end of axis `i`.
value (optional) : T
(Optional) A scalar or rank 1 tensor containing a single value to be filled if the mode chosen is `constant` (by default it is 0.0).

Outputs#

output : T
Tensor after padding.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double)
Constrain input and output types to float tensors.

com.microsoft.QAttention#

Quantization of Multi-Head Self Attention.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

num_heads : int (required)
Number of attention heads
unidirectional : int
Whether every token can only attend to previous tokens. Default value is 0.

Inputs (5 - 9)#

input : T1
3D input tensor with shape (batch_size, sequence_length, input_hidden_size)
weight : T2
2D input tensor with shape (input_hidden_size, 3 * hidden_size), hidden_size = num_heads * head_size
bias : T3
1D input tensor with shape (3 * hidden_size)
input_scale : T3
scale of quantized input tensor. It's a scalar, which means a per-tensor/layer quantization.
weight_scale : T3
scale of weight scale. It's a scalar or a 1D tensor, which means a per-tensor/per-column quantization.Its size should be 3 * hidden_size if it is per-column quantization
mask_index (optional) : T4
Attention mask index with shape (batch_size)
input_zero_point (optional) : T1
zero point of quantized input tensor.It's a scalar, which means a per-tensor/layer quantization.
weight_zero_point (optional) : T2
zero point of quantized weight tensor. It's a scalar or a 1D tensor, which means a per-tensor/per-column quantization.Its size should be 3 * hidden_size if it is per-column quantization
past (optional) : T3
past state for key and value with shape (2, batch_size, num_heads, past_sequence_length, head_size).

Outputs (1 - 2)#

output : T3
3D output tensor with shape (batch_size, sequence_length, hidden_size)
present (optional) : T3
present state for key and value with shape (2, batch_size, num_heads, past_sequence_length + sequence_length, head_size)

Type Constraints#

T1 : tensor(int8), tensor(uint8)
Constrain input and output types to int8 tensors.
T2 : tensor(int8), tensor(uint8)
Constrain input and output types to int8 tensors.
T3 : tensor(float), tensor(float16)
Constrain input and output types to float tensors.
T4 : tensor(int32)
Constrain mask index to integer types

com.microsoft.QGemm#

Quantized Gemm

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

alpha : float
Scalar multiplier for the product of input tensors A * B.
transA : int
Whether A should be transposed
transB : int
Whether B should be transposed

Inputs (6 - 9)#

A : TA
Input tensor A. The shape of A should be (M, K) if transA is 0, or (K, M) if transA is non-zero.
a_scale : T
Scale of quantized input 'A'. It is a scalar,which means a per-tensor quantization.
a_zero_point : TA
Zero point tensor for input 'A'. It is a scalar.
B : TB
Input tensor B. The shape of B should be (K, N) if transB is 0, or (N, K) if transB is non-zero.
b_scale : T
Scale of quantized input 'B'. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
b_zero_point : TB
Zero point tensor for input 'B'. It's optional and default value is 0. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
C (optional) : TC
Optional input tensor C. If not specified, the computation is done as if C is a scalar 0. The shape of C should be unidirectional broadcastable to (M, N). Its type is int32_t and must be quantized with zero_point = 0 and scale = alpha / beta * a_scale * b_scale.
y_scale (optional) : T
Scale of output 'Y'. It is a scalar, which means a per-tensor quantization. It is optional. The output is full precision(float32) if it is not provided. Or the output is quantized.
y_zero_point (optional) : TYZ
Zero point tensor for output 'Y'. It is a scalar, which means a per-tensor quantization. It is optional. The output is full precision(float32) if it is not provided. Or the output is quantized.

Outputs#

Y : TY
Output tensor of shape (M, N).

Type Constraints#

T : tensor(float)
Constrain scale types to float tensors.
TA : tensor(uint8), tensor(int8)
Constrain input A and its zero point types to 8 bit tensors.
TB : tensor(uint8), tensor(int8)
Constrain input B and its zero point types to 8 bit tensors.
TC : tensor(int32)
Constrain input C to 32 bit integer tensors.
TYZ : tensor(uint8), tensor(int8)
Constrain output zero point types to 8 bit tensors.
TY : tensor(float), tensor(uint8), tensor(int8)
Constrain output type to float32 or 8 bit tensors.

com.microsoft.QLinearAdd#

Performs element-wise binary addition on 8 bit data types (with Numpy-style broadcasting support).

C = (A_scale * (A - A_zero_point) + B_scale * (B - B_zero_point))/C_scale + C_zero_point

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs (7 - 8)#

A : T
First operand.
A_scale : tensor(float)
Input A's scale. It's a scalar, which means a per-tensor/layer quantization.
A_zero_point (optional) : T
Input A zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
B : T
Second operand.
B_scale : tensor(float)
Input B's scale. It's a scalar, which means a per-tensor/layer quantization.
B_zero_point (optional) : T
Input B zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
C_scale : tensor(float)
Output scale. It's a scalar, which means a per-tensor/layer quantization.
C_zero_point (optional) : T
Output zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.

Outputs#

C : T
Result, has same element type as two inputs

Type Constraints#

T : tensor(uint8), tensor(int8)
Constrain input and output types to 8 bit signed and unsigned tensors.

com.microsoft.QLinearAveragePool#

QLinearAveragePool consumes an input tensor X and applies average pooling across the tensor according to kernel sizes, stride sizes, and pad lengths. average pooling consisting of computing the average on all values of a subset of the input tensor according to the kernel size and downsampling the data into the output tensor Y for further processing. The output spatial shape will be following:

output_spatial_shape[i] = floor((input_spatial_shape[i] + pad_shape[i] - kernel_spatial_shape[i]) / strides_spatial_shape[i] + 1)

or

output_spatial_shape[i] = ceil((input_spatial_shape[i] + pad_shape[i] - kernel_spatial_shape[i]) / strides_spatial_shape[i] + 1)

if ceil_mode is enabled

* pad_shape[i] is sum of pads along axis i

auto_pad is a DEPRECATED attribute. If you are using them currently, the output spatial shape will be following:

VALID: output_spatial_shape[i] = ceil((input_spatial_shape[i] - kernel_spatial_shape[i] + 1) / strides_spatial_shape[i])
SAME_UPPER or SAME_LOWER: output_spatial_shape[i] = ceil(input_spatial_shape[i] / strides_spatial_shape[i])

And pad shape will be following if SAME_UPPER or SAME_LOWER:

pad_shape[i] = (output_spatial_shape[i] - 1) * strides_spatial_shape[i] + kernel_spatial_shape[i] - input_spatial_shape[i]

The output of each pooling window is divided by the number of elements (exclude pad when attribute count_include_pad is zero).

Input and output scales and zero points are used to convert the output to a new quantization range. Output = Dequantize(Input) -> AveragePool on fp32 data -> Quantize(output)

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

auto_pad : string
auto_pad must be either NOTSET, SAME_UPPER, SAME_LOWER or VALID. Where default value is NOTSET, which means explicit padding is used. SAME_UPPER or SAME_LOWER mean pad the input so that the output spatial size match the input.In case of odd number add the extra padding at the end for SAME_UPPER and at the beginning for SAME_LOWER. VALID mean no padding.
ceil_mode : int
Whether to use ceil or floor (default) to compute the output shape.
channels_last : int
Works on NHWC layout or not? Default not.
count_include_pad : int
Whether include pad pixels when calculating values for the edges. Default is 0, doesn't count include pad.
kernel_shape : list of ints (required)
The size of the kernel along each axis.
pads : list of ints
Padding for the beginning and ending along each spatial axis, it can take any value greater than or equal to 0. The value represent the number of pixels added to the beginning and end part of the corresponding axis. `pads` format should be as follow [x1_begin, x2_begin...x1_end, x2_end,...], where xi_begin the number of pixels added at the beginning of axis `i` and xi_end, the number of pixels added at the end of axis `i`. This attribute cannot be used simultaneously with auto_pad attribute. If not present, the padding defaults to 0 along start and end of each spatial axis.
strides : list of ints
Stride along each spatial axis. If not present, the stride defaults to 1 along each spatial axis.

Inputs (4 - 5)#

X : T
Input data tensor from the previous operator; dimensions for image case are (N x C x H x W), where N is the batch size, C is the number of channels, and H and W are the height and the width of the data. For non image case, the dimensions are in the form of (N x C x D1 x D2 ... Dn), where N is the batch size. Optionally, if dimension denotation is in effect, the operation expects the input data tensor to arrive with the dimension denotation of [DATA_BATCH, DATA_CHANNEL, DATA_FEATURE, DATA_FEATURE ...].
x_scale : tensor(float)
Input scale. It's a scalar, which means a per-tensor/layer quantization.
x_zero_point (optional) : T
Input zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
y_scale : tensor(float)
Output scale. It's a scalar, which means a per-tensor/layer quantization.
y_zero_point (optional) : T
Output zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.

Outputs#

Y : T
Output data tensor from average or max pooling across the input tensor. Dimensions will vary based on various kernel, stride, and pad sizes. Floor value of the dimension is used

Type Constraints#

T : tensor(uint8), tensor(int8)
Constrain input and output types to 8 bit tensors.

com.microsoft.QLinearConcat#

Concatenate a list of tensors into a single tensor.All input tensors must have the same shape, except for the dimension size of the axis to concatenate on.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

axis : int (required)
Which axis to concat on

Inputs (3 - ∞)#

Y_scale : TF
Y's scale.
Y_zero_point : T8
Y's zero point.
inputs (variadic, heterogeneous) : TV
List of tensors/scale/zero_point for concatenation

Outputs#

Y : T8
Concatenated tensor

Type Constraints#

T8 : tensor(uint8), tensor(int8)
Constrain input and output types to 8 bit signed and unsigned tensors.
TF : tensor(float)
Constrain scale types to any float tensor type.
TV : tensor(uint8), tensor(int8), tensor(float)
Sequence of (Tensor, Scale, ZeroPoint) tuples. The type is sequence of (T8, TF, T8).

com.microsoft.QLinearConv#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

auto_pad : string
channels_last : int
dilations : list of ints
group : int
kernel_shape : list of ints
pads : list of ints
strides : list of ints

Inputs (8 - 9)#

x : T1
x_scale : tensor(float)
x_zero_point : T1
w : T2
w_scale : tensor(float)
w_zero_point : T2
y_scale : tensor(float)
y_zero_point : T3
B (optional) : T4

Outputs#

y : T3

Type Constraints#

T1 : tensor(int8), tensor(uint8)
T2 : tensor(int8), tensor(uint8)
T3 : tensor(int8), tensor(uint8)
T4 : tensor(int32)

com.microsoft.QLinearGlobalAveragePool#

QLinearGlobalAveragePool consumes an input tensor X and applies Average pooling across the values in the same channel. This is equivalent to AveragePool with kernel size equal to the spatial dimension of input tensor. Input is of type uint8_t or int8_t.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

channels_last : int

Inputs#

X : T
Input data tensor from the previous operator; According to channels_last, dimensions for image case are (N x C x H x W), or (N x H x W x C) where N is the batch size, C is the number of channels, and H and W are the height and the width of the data. For non image case, the dimensions are in the form of (N x C x D1 x D2 ... Dn), or (N x D1 X D2 ... Dn x C) where N is the batch size.
x_scale : tensor(float)
Scale of quantized input 'X'. It must be a scalar.
x_zero_point : T
Zero point tensor for input 'X'. It must be a scalar.
y_scale : tensor(float)
Scale of quantized output 'Y'. It must be a scalar.
y_zero_point : T
Zero point tensor for output 'Y'. It must be a scalar.

Outputs#

Y : T
Output data tensor from pooling across the input tensor. The output tensor has the same rank as the input. with the N and C value keep it value, while the otherdimensions are all 1.

Type Constraints#

T : tensor(uint8), tensor(int8)
Constrain input and output types to signed/unsigned int8 tensors.

com.microsoft.QLinearLeakyRelu#

QLinearLeakyRelu takes quantized input data (Tensor), an argument alpha, and quantize parameter for output, and produces one output data (Tensor) where the function f(x) = quantize(alpha * dequantize(x)) for dequantize(x) < 0, f(x) = quantize(dequantize(x)) for dequantize(x) >= 0, is applied to the data tensor elementwise.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

alpha : float
Coefficient of leakage.

Inputs (4 - 5)#

X : T
Input tensor
X_scale : tensor(float)
Input X's scale. It's a scalar, which means a per-tensor/layer quantization.
X_zero_point (optional) : T
Input X's zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
Y_scale : tensor(float)
Output Y's scale. It's a scalar, which means a per-tensor/layer quantization.
Y_zero_point (optional) : T
Output Y's zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.

Outputs#

Y : T
Output tensor

Type Constraints#

T : tensor(uint8), tensor(int8)
Constrain input and output types to 8 bit tensors.

com.microsoft.QLinearMul#

Performs element-wise binary multiplication on 8 bit data types (with Numpy-style broadcasting support).

C = ((A - A_zero_point) * (B - B_zero_point)) * (A_scale * B_scale)/C_scale + C_zero_point

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs (7 - 8)#

A : T
First operand.
A_scale : tensor(float)
Input A's scale. It's a scalar, which means a per-tensor/layer quantization.
A_zero_point (optional) : T
Input A zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
B : T
Second operand.
B_scale : tensor(float)
Input B's scale. It's a scalar, which means a per-tensor/layer quantization.
B_zero_point (optional) : T
Input B zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
C_scale : tensor(float)
Output scale. It's a scalar, which means a per-tensor/layer quantization.
C_zero_point (optional) : T
Output zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.

Outputs#

C : T
Result, has same element type as two inputs

Type Constraints#

T : tensor(uint8), tensor(int8)
Constrain input and output types to 8 bit signed and unsigned tensors.

com.microsoft.QLinearReduceMean#

Computes the mean of the low-precision input tensor’s element along the provided axes. The resulting tensor has the same rank as the input if keepdims equal 1. If keepdims equal 0, then the resulting tensor have the reduced dimension pruned. The above behavior is similar to numpy, with the exception that numpy default keepdims to False instead of True. Input and Output scales and zero points are used to requantize the output in a new range. This helps to improve accuracy as after ReduceMean operation the range of the output is expected to decrease.

"Output = Dequantize(Input) -> ReduceMean on fp32 data -> Quantize(output)",

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

axes : list of ints (required)
A list of integers, along which to reduce. The default is to reduce over all the dimensions of the input tensor.
keepdims : int (required)
Keep the reduced dimension or not, default 1 mean keep reduced dimension.

Inputs (4 - 5)#

data : T
An input tensor.
data_scale : tensor(float)
Input scale. It's a scalar, which means a per-tensor/layer quantization.
data_zero_point (optional) : T
Input zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
reduced_scale : tensor(float)
Output scale. It's a scalar, which means a per-tensor/layer quantization.
reduced_zero_point (optional) : T
Output zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.

Outputs#

reduced : T
Reduced output tensor.

Type Constraints#

T : tensor(uint8), tensor(int8)
Constrain input types to 8 bit signed and unsigned tensors.

com.microsoft.QLinearSigmoid#

QLinearSigmoid takes quantized input data (Tensor), and quantize parameter for output, and produces one output data (Tensor) where the function f(x) = quantize(Sigmoid(dequantize(x))), is applied to the data tensor elementwise. Wwhere the function Sigmoid(x) = 1 / (1 + exp(-x))

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs (4 - 5)#

X : T
Input tensor
X_scale : tensor(float)
Input X's scale. It's a scalar, which means a per-tensor/layer quantization.
X_zero_point (optional) : T
Input X's zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
Y_scale : tensor(float)
Output Y's scale. It's a scalar, which means a per-tensor/layer quantization.
Y_zero_point (optional) : T
Output Y's zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.

Outputs#

Y : T
Output tensor

Type Constraints#

T : tensor(uint8), tensor(int8)
Constrain input and output types to 8 bit tensors.

com.microsoft.QLinearSoftmax#

QLinearSoftmax computes the normalized exponential values for the given input: Softmax(input, axis) = Exp(input) / ReduceSum(Exp(input), axis=axis, keepdims=1) The input does not need to explicitly be a 2D vector. The “axis” attribute indicates the dimension along which QLinearSoftmax will be performed for onnx v.13+. or the dimension coerced to NxD Matrix for onnx v.12-. The output tensor has the same shape.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

axis : int
apply softmax to elements for dimensions axis,or all dims along with axis according to op-version
opset : int (required)
opset version of corresponding SoftMax.

Inputs#

X : T
The input tensor
X_scale : tensor(float)
Scale of quantized input 'X'. It must be a scalar.
x_zero_point (optional) : T
Zero point tensor for input 'X'.It must be a scalar.
y_scale : tensor(float)
Scale of quantized output 'Y'. It must be a scalar.
y_zero_point : T
Zero point tensor for output 'Y'. It must be a scalar.

Outputs#

Y : T
Output data tensor from pooling across the input tensor. The output tensor has the same rank as the input.

Type Constraints#

T : tensor(uint8), tensor(int8)
Constrain input and output types to signed/unsigned int8 tensors.

com.microsoft.QOrderedAttention#

Quantized version of simplified Multi-Head Self Attention(using int8 with specific matrix Layout). Multi-Head Self Attention that can be either unidirectional (like GPT-2) or bidirectional (like BERT). The mask_index input is optional. Besides raw attention mask with shape (batch_size, past_sequence_length + sequence_length) or (batch_size, sequence_length, past_sequence_length + sequence_length) with value 0 for masked and 1 otherwise, we also support other two formats: When input has right-side padding, mask_index is one dimension with shape (batch_size), where value of each element is the end position, or valid length of actual sequence excluding padding. When input has left-side padding, mask_index has shape (2 * batch_size), where the values are the exclusive end positions followed by the inclusive start positions. When unidirectional is 1, and each token only attend to previous tokens. For GPT-2, both past and present state are optional. Present state could appear in output even when past state is not in input. Current version does not support past/present, extra_add and qkv_hidden_sizes. TODO: Support them if needed in the future.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

num_heads : int (required)
Number of attention heads
order_input : int (required)
cublasLt order of input matrix. See the schema of QuantizeWithOrder for order definition.
order_output : int (required)
cublasLt order of global bias
order_weight : int (required)
cublasLt order of weight matrix
qkv_hidden_sizes : list of ints
Hidden layer sizes of Q, K, V paths in Attention
unidirectional : int
Whether every token can only attend to previous tokens. Default value is 0.

Inputs (17 - 20)#

input : Q
3D input tensor with shape (batch_size, sequence_length, input_hidden_size)
scale_input : S
scale of the input, scalar value (per tensor) currently.
scale_Q_gemm : S
scale of the gemm - scalar (per-tensor quantization)
scale_K_gemm : S
scale of the gemm - scalar (per-tensor quantization)
scale_V_gemm : S
scale of the gemm - scalar (per-tensor quantization)
Q_weight : Q
2D input tensor with shape (input_hidden_size, hidden_size), where hidden_size = num_heads * head_size
K_weight : Q
2D input tensor with shape (input_hidden_size, hidden_size), where hidden_size = num_heads * head_size
V_weight : Q
2D input tensor with shape (input_hidden_size, hidden_size), where hidden_size = num_heads * head_size
scale_Q_weight : S
scale of the weight (scalar for per-tensor quantization or 1-D of dims [hidden_size] for per-channel quantization)
scale_K_weight : S
scale of the weight (scalar for per-tensor quantization or 1-D of dims [hidden_size] for per-channel quantization)
scale_V_weight : S
scale of the weight (scalar for per-tensor quantization or 1-D of dims [hidden_size] for per-channel quantization)
Q_bias : S
1D input tensor with shape (hidden_size)
K_bias : S
1D input tensor with shape (hidden_size)
V_bias : S
1D input tensor with shape (hidden_size)
scale_QKT_gemm (optional) : S
scale of the gemm - scalar (per-tensor quantization)
scale_QKT_softmax (optional) : S
scale of the softmax result - scalar (per-tensor quantization)
scale_values_gemm : S
scale of the gemm - scalar (per-tensor quantization). Also this is the output scale for the operator.
mask_index (optional) : G
Attention mask with shape (batch_size, 1, max_sequence_length, max_sequence_length), (batch_size, past_sequence_length + sequence_length)or (batch_size, sequence_length, past_sequence_length + sequence_length), or index with shape (batch_size) or (2 * batch_size).
past (optional) : Q
past state for key and value with shape (2, batch_size, num_heads, past_sequence_length, head_size).
extra_add (optional) : S
additional add to QxK' with shape (batch_size, num_heads, sequence_length, sequence_length).

Outputs#

output : Q
3D output tensor with shape (batch_size, sequence_length, hidden_size)

Type Constraints#

Q : tensor(int8)
Constrain input and output types to int8 tensors.
S : tensor(float)
Constrain scales to float32 tensors.
G : tensor(int32)
Constrain to integer types

com.microsoft.QOrderedGelu#

Ordered Quantize Gelu.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

order_X : int
cublasLt order of input X. Optional. See the schema of QuantizeWithOrder for order definition.
order_Y : int
cublasLt order of matrix Y, must be same as order_X if specified together. Optional.

Inputs#

X : Q
N-dimensional input A
scale_X : S
scale of the input A
scale_Y : S
scale of the output Y

Outputs#

Y : Q
Output of the Gelu

Type Constraints#

Q : tensor(int8)
Constrain input and output types to int8 tensors.
S : tensor(float)
Constrain scales to float32

com.microsoft.QOrderedLayerNormalization#

QOrderedLayerNormalization

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

axis : int
The first normalization dimension: normalization will be performed along dimensions axis : rank(inputs).
epsilon : float
The epsilon value to use to avoid division by zero.
order_X : int
cublasLt order of input X. Default is ROW MAJOR. See the schema of QuantizeWithOrder for order definition.
order_Y : int
cublasLt order of matrix Y, must be same as order_X. Default is ROW MAJOR.

Inputs#

X : Q
Input data tensor from the previous layer.
scale_X : S
scale of the quantized X
scale : F
Scale tensor, i.e., gamma vector.
B (optional) : F
Bias tensor.
scale_Y : S
scale of the quantized X

Outputs#

Y : Q
Output data tensor.

Type Constraints#

F : tensor(float16), tensor(float)
Constrain input gamma and bias could be float16/float tensors. float may get better precision, float16 runs faster.
S : tensor(float)
quantization scale must be float tensors.
Q : tensor(int8)
quantization tensor must be int8 tensors.

com.microsoft.QOrderedLongformerAttention#

Quantized version of Longformer Self Attention (using int8 with specific matrix Layout).

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

num_heads : int (required)
Number of attention heads
order_global_weight : int (required)
cublasLt order of weight matrix
order_input : int (required)
cublasLt order of input matrix. See the schema of QuantizeWithOrder for order definition.
order_output : int (required)
cublasLt order of global bias
order_weight : int (required)
cublasLt order of weight matrix
window : int (required)
One sided attention windows length W, or half of total window length

Inputs#

input : Q
3D input tensor with shape (batch_size, sequence_length, hidden_size), hidden_size = num_heads * head_size
scale_input : S
scale of the input
weight : Q
2D input tensor with shape (hidden_size, 3 * hidden_size)
scale_weight : S
scale of the weight
bias : S
1D input tensor with shape (3 * hidden_size), fp32 only currently.
scale_bias : S
reserved. (not used as add bias need float value in cublasLt for normal order.)
scale_qkv_gemm : S
scale of the output for fused kqv gemm
mask : F
Attention mask with shape (batch_size, sequence_length)
global_weight : Q
2D input tensor with shape (hidden_size, 3 * hidden_size)
scale_global_weight : S
scale of the global_weight
global_bias : S
1D input tensor with shape (3 * hidden_size)
scale_global_gemm : S
scale of the global_qkv_gemm
global : G
Global attention flags with shape (batch_size, sequence_length)
scale_output : S
scale of the output

Outputs#

output : Q
3D output tensor with shape (batch_size, sequence_length, hidden_size)

Type Constraints#

Q : tensor(int8)
Constrain input and output types to int8 tensors.
S : tensor(float)
Constrain scales to float32 tensors.
G : tensor(int32)
Constrain to integer types
F : tensor(float16)
Be compatible with float version.

com.microsoft.QOrderedMatMul#

Quantize (Int8) MatMul with order. Implement Y = alpha * A * B + bias + beta * C. Matrix A, B, C, Y are all int8 matrix. Two type of order combination supported: *) When order_B is ORDER_COL, order_A must be ORDER_ROW. bias is vector of {#cols of Y} of float32, C should be batch 1/batch_A. B could be of batch 1 or batch_A. Note B is reorder to ORDER_COL, or Transposed. Not Transposed first and then Reordered here. *) When order_B is specify ORDER_COL4_4R2_8C or ORDER_COL32_2R_4R4, orderA must be ORDER_COL32. MatMul will be implemented using alpha(A * B) + beta * C => Y. bias is not supported here. B in fact is transposed first then reordered into ORDER_COL4_4R2_8C or ORDER_COL32_2R_4R4 here. order_Y and order_C will be same as order_A. Support per column quantized weight, ie, scale_B is 1-D vector of size [#cols of matrix B].

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

order_A : int (required)
cublasLt order of matrix A. See the schema of QuantizeWithOrder for order definition.
order_B : int (required)
cublasLt order of matrix B
order_Y : int (required)
cublasLt order of matrix Y and optional matrix C

Inputs (5 - 8)#

A : Q
3-dimensional matrix A
scale_A : S
scale of the input A.
B : Q
2-dimensional matrix B. Transposed if order_B is ORDER_COL.
scale_B : S
scale of the input B. Scalar or 1-D float32.
scale_Y : S
scale of the output Y.
bias (optional) : S
1d bias, not scaled with scale_Y.
C (optional) : Q
3d or 2d matrix C. if 2d expand to 3d first. Shape[0] should be 1 or same as A.shape[0]
scale_C (optional) : S
scale of the input A.

Outputs#

Y : Q
Matrix multiply results from A * B

Type Constraints#

Q : tensor(int8)
Constrain input and output types to int8 tensors.
S : tensor(float)
Constrain bias and scales to float32

com.microsoft.QuantizeBFP#

The BFP quantization operator. It consumes a full precision tensor and computes an BFP tensor. More documentation on the BFP format can be found in this paper: https://www.microsoft.com/en-us/research/publication/pushing-the-limits-of-narrow-precision-inferencing-at-cloud-scale-with-microsoft-floating-point/

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

bfp_type : int (required)
The type of BFP - must match with the BFPType enum
block_dim : int
Each bounding box spans this dimension.Typically, the block dimension corresponds to the reduction dimension of the matrix multipication that consumes the output of this operator.For example, for a 2D matrix multiplication A@W, QuantizeBFP(A) would use block_dim 1 and QuantizeBFP(W) would use block_dim 0.The default is the last dimension.

Inputs#

x : T1
N-D full precision input tensor to be quantized.

Outputs#

y : T2
1-D, contiguous BFP data
shape : T3
Shape of x
strides : T3
Strides of x

Type Constraints#

T1 : tensor(float), tensor(float16), tensor(bfloat16)
Constrain the input to float and bfloat.
T2 : tensor(uint8)
Constrain y to uint8.
T3 : tensor(int64)
Constrain shape and strides to uint64.

com.microsoft.QuantizeLinear#

The linear quantization operator. It consumes a full precision data, a scale, a zero point to compute the low precision / quantized tensor. The quantization formula is y = saturate ((x / y_scale) + y_zero_point).For saturation, it saturates to [0, 255] if it’s uint8, or [-128, 127] if it’s int8. For (x / y_scale), it’s rounding to nearest ties to even. Refer to https://en.wikipedia.org/wiki/Rounding for details. Scale and zero point must have same shape. They must be either scalar (per tensor) or 1-D tensor (per ‘axis’).

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

axis : int
The axis along which same quantization parameters are applied. It's optional.If it's not specified, it means per-tensor quantization and input 'x_scale' and 'x_zero_point' must be scalars.If it's specified, it means per 'axis' quantization and input 'x_scale' and 'x_zero_point' must be 1-D tensors.

Inputs#

x : T1
N-D full precision Input tensor to be quantized.
y_scale : T1
Scale for doing quantization to get 'y'. It could be a scalar or a 1-D tensor,which means a per-tensor or per-axis quantization. If it's a 1-D tensor, its number of elements should be equal to the dimension value of 'axis' dimension of input 'x'.
y_zero_point : T2
Zero point for doing quantization to get 'y'. It could be a scalar or a 1-D tensor, which means a per-tensoror per-axis quantization. If it's a 1-D tensor, its number of elements should be equal to the dimension value of 'axis' dimension of input 'x'.

Outputs#

y : T2
N-D quantized output tensor. It has same shape as input 'x'.

Type Constraints#

T1 : tensor(float16), tensor(float)
Constrain 'x', 'y_scale' to float tensors.
T2 : tensor(int8), tensor(uint8)
Constrain 'y_zero_point' and 'y' to 8-bit integer tensors.

com.microsoft.QuantizeWithOrder#

Quantize input matrix to specific layout used in cublaslt.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

order_input : int (required)
cublasLt order of input matrix. ORDER_COL = 0, ORDER_ROW = 1, ORDER_COL32 = 2, ORDER_COL4_4R2_8C = 3, ORDER_COL32_2R_4R4 = 4. Please refer https://docs.nvidia.com/cuda/cublas/index.html#cublasLtOrder_t for their meaning.
order_output : int (required)
cublasLt order of output matrix.

Inputs#

input : F
TODO: input tensor of (ROWS, COLS). if less than 2d, will broadcast to (1, X). If 3d, it is treated as (B, ROWS, COS)
scale_input : S
scale of the input

Outputs#

output : Q
output tensor

Type Constraints#

Q : tensor(int8)
Constrain input and output types to int8 tensors.
F : tensor(float16), tensor(float)
Constrain to float types
S : tensor(float)
Constrain Scale to float32 types

com.microsoft.QuickGelu#

Compute x * Sigmoid(alpha * x).

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

alpha : float
Alpha value.

Inputs#

X : T
The input data as Tensor.

Outputs#

Y : T
The output.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input and output types to float tensors.

com.microsoft.Range#

Creates a sequence of numbers that begins at start and extends by increments of delta up to but not including limit.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs (2 - 3)#

start : T
Tensor(scalar, or dims=[1]). First entry in the range.
limit : T
Tensor(scalar, or dims=[1]). Upper limit of sequence, exclusive.
delta (optional) : T
Tensor(scalar, or dims=[1]). Number that increments start. Defaults to 1.

Outputs#

Y : T
1-D Tensor of the range.

Type Constraints#

T : tensor(float), tensor(double), tensor(int16), tensor(int32), tensor(int64)
Constrain input and output types.

com.microsoft.ReduceSumInteger#

Computes the sum of the low-precision input tensor’s element along the provided axes. The resulting tensor has the same rank as the input if keepdims equal 1. If keepdims equal 0, then the resulting tensor have the reduced dimension pruned. The above behavior is similar to numpy, with the exception that numpy default keepdims to False instead of True.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

axes : list of ints (required)
A list of integers, along which to reduce. The default is to reduce over all the dimensions of the input tensor.
keepdims : int (required)
Keep the reduced dimension or not, default 1 mean keep reduced dimension.

Inputs#

data : T1
An input tensor.

Outputs#

reduced : T2
Reduced output tensor.

Type Constraints#

T1 : tensor(int8), tensor(uint8)
Constrain input type to 8-bit integer tensor.
T2 : tensor(int32), tensor(uint32)
Constrain output data type to 32-bit integer tensor.T2 must be tensor(uint32) when T1 is tensor(uint8),or must be tensor(int32) when T1 is tensor(int8).

com.microsoft.Rfft#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

normalized : int
onesided : int
signal_ndim : int

Inputs#

X : T
input tensor

Outputs#

Y : T
output tensor

Type Constraints#

T : tensor(float), tensor(double), tensor(float16)
Constrain input and output types to float or half tensors.

com.microsoft.SampleOp#

Sample echo operator.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs#

X : T
input

Outputs#

Y : T
output

Type Constraints#

T : tensor(uint32), tensor(uint64), tensor(int32), tensor(int64), tensor(float16), tensor(float), tensor(double)
Constrain to any tensor type. If the dtype attribute is not provided this must be a valid output type.

com.microsoft.SkipLayerNormalization#

Skip and Layer Normalization Fusion

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

epsilon : float
The epsilon value to use to avoid division by zero.

Inputs (3 - 5)#

input : T
3D input tensor with shape (batch_size, sequence_length, hidden_size)
skip : T
3D skip tensor with shape (batch_size, sequence_length, hidden_size)
gamma : T
1D input tensor with shape (hidden_size)
beta (optional) : T
1D skip tensor with shape (hidden_size
bias (optional) : T
1D bias tensor with shape (hidden_size

Outputs (1 - 3)#

output : T
3D output tensor with shape (batch_size, sequence_length, hidden_size)
mean (optional) : U
Saved mean used during training to speed up gradient computation
inv_std_var (optional) : U
Saved inverse standard variance used during training to speed up gradient computation.

Type Constraints#

T : tensor(float), tensor(float16)
Constrain input and output types to float or half tensors.
U : tensor(float)
Constrain mean and inv_std_var to float tensors.

com.microsoft.Snpe#

Onnx node for SNPE.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

DLC : string (required)
payload of the SNPE DLC file.
notes : string
(Optional) Some notes for the model
snpe_version : string
(Optional) SNPE version used to convert the model.
target_device : string
(Optional) Target device like CPU, DSP, etc.

Inputs (1 - ∞)#

inputs (variadic) : T
List of tensors for SNPE DLC input

Outputs (1 - ∞)#

outputs (variadic) : T
One or more outputs, list of tensors for DLC output

Type Constraints#

T : tensor(uint8), tensor(uint16), tensor(float)
Constrain input and output types to uint8, uint16, float tensors.

com.microsoft.SparseToDenseMatMul#

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

alpha : float
Scalar multiplier for the product of the input tensors.
transA : int
Whether A should be transposed on the last two dimensions before doing multiplication
transB : int
Whether B should be transposed on the last two dimensions before doing multiplication

Inputs#

A : T
2-dimensional sparse matrix A. Either COO or CSR format
B : T1
N-dimensional dense matrix B

Outputs#

Y : T1
Matrix multiply results

Type Constraints#

T : sparse_tensor(float), sparse_tensor(double), sparse_tensor(int64), sparse_tensor(int32), sparse_tensor(uint64), sparse_tensor(uint32)
Constrain input and output types to float tensors.
T1 : tensor(float), tensor(double), tensor(int64), tensor(int32), tensor(uint64), tensor(uint32)
Constrain input and output types to float tensors.

com.microsoft.Tokenizer#

Tokenizer divides each string in X into a vector of strings along the last axis. Allowed input shapes are [C] and [N, C]. If the maximum number of tokens found per input string is D, the output shape would be [N, C, D] when input shape is [N, C]. Similarly, if input shape is [C] then the output should be [C, D]. Tokenizer has two different operation modes. The first mode is selected when “tokenexp” is not set and “separators” is set. If “tokenexp” is set and “separators” is not set, the second mode will be used. The first mode breaks each input string into tokens by matching and removing separators. “separators” is a list of strings which are regular expressions. “tokenexp” is a single regular expression. Let’s assume “separators” is [” “] and consider an example. If input is [“Hello World”, “I love computer science !”] whose shape is [2], then the output would be [[“Hello”, “World”, padvalue, padvalue, padvalue], [“I”, “love”, “computer”, “science”, “!”]] whose shape is [2, 5] because you can find at most 5 tokens per input string. Note that the input at most can have two axes, so 3-D and higher dimension are not supported. If “separators” contains a single empty string, the Tokenizer will enter into character tokenezation mode. This means all strings will be broken part into individual characters. For each input string, the second mode searches matches of “tokenexp” and each match will be a token in Y. The matching of “tokenexp” is conducted greedily (i.e., a match should be as long as possible). This operator searches for the first match starting from the beginning of the considered string, and then launches another search starting from the first remained character after the first matched token. If no match found, this operator will remove the first character from the remained string and do another search. This procedure will be repeated until reaching the end of the considered string. Let’s consider another example to illustrate the effect of setting “mark” to true. If input is [“Hello”, “World”], then the corresponding output would be [0x02, “Hello”, “World”, 0x03]. This implies that if mark is true, [C]/[N, C] - input’s output shape becomes [C, D+2]/[N, C, D+2]. If tokenizer removes the entire content of [C]-input, it will produce [[]]. I.e. the output shape should be [C][0] or [N][C][0] if input shape was [N][C]. If the tokenizer receives empty input of [0] then the output is [0] if empty input of [N, 0] then [N, 0].

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

mark : int (required)
Boolean whether to mark the beginning/end character with start of text character (0x02)/end of text character (0x03).
mincharnum : int (required)
Minimum number of characters allowed in the output. For example, if mincharnum is 2, tokens such as "A" and "B" would be ignored
pad_value : string (required)
The string used to pad output tensors when the tokens extracted doesn't match the maximum number of tokens found. If start/end markers are needed, padding will appear outside the markers.
separators : list of strings
an optional list of strings attribute that contains a list of separators - regular expressions to match separators Two consecutive segments in X connected by a separator would be divided into two tokens. For example, if the input is "Hello World!" and this attribute contains only one space character, the corresponding output would be ["Hello", "World!"]. To achieve character-level tokenization, one should set the 'separators' to [""], which contains an empty string.
tokenexp : string
An optional string. Token's regular expression in basic POSIX format (pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_03). If set, tokenizer may produce tokens matching the specified pattern. Note that one and only of 'tokenexp' and 'separators' should be set.

Inputs#

X : T
Strings to tokenize

Outputs#

Y : T
Tokenized strings

Type Constraints#

T : tensor(string)
Input/Output is a string tensor

com.microsoft.TorchEmbedding#

Based on Torch operator Embedding, creates a lookup table of embedding vectors of fixed size, for a dictionary of fixed size.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs (2 - 4)#

weight : T
The embedding matrix of size N x M. 'N' is equal to the maximum possible index + 1, and 'M' is equal to the embedding size
indices : tensor(int64)
Long tensor containing the indices to extract from embedding matrix.
padding_idx (optional) : tensor(int64)
A 0-D scalar tensor. If specified, the entries at `padding_idx` do not contribute to the gradient; therefore, the embedding vector at `padding_idx` is not updated during training, i.e. it remains as a fixed pad.
scale_grad_by_freq (optional) : tensor(bool)
A 0-D bool tensor. If given, this will scale gradients by the inverse of frequency of the indices (words) in the mini-batch. Default is ``False``

Outputs#

Y : T
Output tensor of the same type as the input tensor. Shape of the output is * x M, where '*' is the shape of input indices, and 'M' is the embedding size.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16), tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64)
Constrain input and output types to all numeric tensors.

com.microsoft.TransposeMatMul#

Duplicate of FusedMatMul. Going forward FusedMatMul should be used. This OP will be supported for backward compatibility. Matrix product that behaves like numpy.matmul: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.matmul.html

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

alpha : float
Scalar multiplier for the product of the input tensors.
transA : int
Whether A should be transposed on the last two dimensions before doing multiplication
transB : int
Whether B should be transposed on the last two dimensions before doing multiplication

Inputs#

A : T
N-dimensional matrix A
B : T
N-dimensional matrix B

Outputs#

Y : T
Matrix multiply results

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input and output types to float tensors.

com.microsoft.Trilu#

Returns the upper or lower triangular part of a 2-D matrix, or batches of 2-D matrices. If the attribute “upper” is set to true, the upper triangular matrix is retained. Lower triangular matrix is retained otherwise. Default value for upper is true. Trilu takes one input tensor of shape [*, N, M], where * is zero or more batch dimensions. The upper triangular part consists of the elements on and above the given diagonal (k). The lower triangular part consists of elements on and below the diagonal. All other elements in the matrix are set to zero. If k = 0, the triangular part on and above/below the main diagonal is retained. If upper is set to true, a positive k retains the upper triangular matrix excluding k diagonals above the main diagonal. A negative k value includes as many diagonals below the main diagonal. If upper is set to false, a positive k retains the lower triangular matrix including k diagonals above the main diagonal. A negative k value excludes as many diagonals below the main diagonal.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

upper : int
Boolean. Indicates whether upper or lower part of matrix is retained. Default is true.

Inputs (1 - 2)#

X : T
Input tensor of rank 2 or higher.
k (optional) : tensor(int64)
A 0-D tensor containing a single value corresponding to the number diagonals above or the main diagonal to exclude or include.Default value is 0 if it's not specified.

Outputs#

Y : T
Output tensor of the same type and shape as the input tensor.

Type Constraints#

T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16), tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(bool)
Constrain input and output types to all numeric tensors and bool tensors.

com.microsoft.Unique#

Finds all the unique values (deduped list) present in the given input tensor. This operator returns 3 outputs. The first output tensor ‘uniques’ contains all of the unique elements of the input, sorted in the same order that they occur in the input. The second output tensor ‘idx’ is the same size as the input and it contains the index of each value of the input in ‘uniques’. The third output tensor ‘counts’ contains the count of each element of ‘uniques’ in the input. Example: input_x = [2, 1, 1, 3, 4, 3] output_uniques = [2, 1, 3, 4] output_idx = [0, 1, 1, 2, 3, 2] output_counts = [1, 2, 2, 1]

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Inputs#

x : T
A 1-D input tensor that is to be processed.

Outputs#

y : T
A 1-D tensor of the same type as 'x' containing all the unique values in 'x' sorted in the same order that they occur in the input 'x'
idx : tensor(int64)
A 1-D INT64 tensor of the same size as 'x' containing the indices for each value in 'x' in the output 'uniques'
counts : tensor(int64)
A 1-D INT64 tensor containing the the count of each element of 'uniques' in the input 'x'

Type Constraints#

T : tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(float16), tensor(float), tensor(double), tensor(string), tensor(bool), tensor(complex64), tensor(complex128)
Input can be of any tensor type.

com.microsoft.WordConvEmbedding#

The WordConvEmbedding takes in a batch of sequence words and embed each word to a vector.

Version#

This version of the operator has been available since version 1 of the ‘com.microsoft’ operator set.

Attributes#

char_embedding_size : int
Integer representing the embedding vector size for each char.If not provide, use the char embedding size of embedding vector.
conv_window_size : int
This operator applies convolution to word from left to right with window equal to conv_window_size and stride to 1.Take word 'example' for example, with conv_window_size equal to 2, conv is applied to [ex],[xa], [am], [mp]...If not provide, use the first dimension of conv kernel shape.
embedding_size : int
Integer representing the embedding vector size for each word.If not provide, use the filter size of conv weight

Inputs#

Sequence : T
Specify batchs of sequence words to embedding
W : T1
Specify weights of conv
B : T1
Specify bias of conv
C : T1
Specify embedding vector of char

Outputs#

Y : T1
output

Type Constraints#

T : tensor(int32)
Constrain to tensor(int32).
T1 : tensor(float)
Constrain to tensor(float).

experimental com.microsoft.IsAllFinite#

IsAllFinite

Version#

No versioning maintained for experimental ops.

Attributes#

isinf_only : int
If true, check only for Inf, -Inf.
isnan_only : int
If true, check only for NaN.

Inputs (1 - ∞)#

input (variadic) : V
Input tensors to check.

Outputs#

output : T
The output scalar. Its value is true if all input tensors are finite. Otherwise, the output value would be false.

Type Constraints#

V : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input and output types to float tensors.
T : tensor(bool)
Constrain the output to a boolean tensor.

experimental com.microsoft.QEmbedLayerNormalization#

QEmbedLayerNormalization is the quantized fusion of embedding layer in BERT model, with optional mask processing. The embedding layer takes input_ids (word IDs) and segment_ids (sentence IDs) to look up word_embedding, position_embedding, and segment_emedding; the embeddings are added then applied layer normalization using gamma and beta tensors. The input_ids and segment_ids remain int32. All embeddings, gamma, and beta tensors are converted to int8/uint8. The last input mask is optional. If mask is provided, mask index (that is position of first 0 in mask, or number of words will be calculated.

Version#

No versioning maintained for experimental ops.

Attributes#

epsilon : float
The epsilon value to use to avoid division by zero.

Inputs#

input_ids : T1
2D words IDs with shape (batch_size, sequence_length)
segment_ids (optional) : T1
2D segment IDs with shape (batch_size, sequence_length)
word_embedding_quant : T2
2D with shape (,hidden_size)
position_embedding_quant : T2
2D with shape (, hidden_size)
segment_embedding (optional) : T2
2D with shape (, hidden_size)
gamma_quant : T2
1D gamma tensor for layer normalization with shape (hidden_size)
beta_quant : T2
1D beta tensor for layer normalization with shape (hidden_size)
mask (optional) : T1
Mask
word_embedding_scale : T
Scale for word embeddings
position_embedding_scale : T
Scale for position embeddings
segment_embedding_scale (optional) : T
Scale for segment embeddings
gamma_scale : T
Scale for 1D gamma tensor
beta_scale : T
Scale for 1D beta tensor
word_embedding_zero_point : T2
Zero point for word embeddings
position_embedding_zero_point : T2
Zero point for position embeddings
segment_embedding_zero_point (optional) : T2
Zero Point for segment embeddings
gamma_zero_point : T2
Zero Point for 1D gamma tensor
beta_zero_point : T2
Zero Point for 1D beta tensor

Outputs#

layernorm_out : T
LayerNorm Output
mask_index_out : T1
Mask Index Output

Type Constraints#

T1 : tensor(int32)
Constrain mask index to integer types
T2 : tensor(int8), tensor(uint8)
Constrain input and output types to int8 tensors.
T : tensor(float)
Constrain input and output types to float32 tensors.