Type Denotation#

Type Denotation is used to describe semantic information around what the inputs and outputs are. It is stored on the TypeProto message.

Motivation#

The motivation of such a mechanism can be illustrated via a simple example. In the neural network SqueezeNet, it takes in an NCHW image input float[1,3,244,244] and produces a output float[1,1000,1,1]:

input_in_NCHW -> data_0 -> SqueezeNet() -> output_softmaxout_1

In order to run this model the user needs a lot of information. In this case the user needs to know:

  • the input is an image

  • the image is in the format of NCHW

  • the color channels are in the order of bgr

  • the pixel data is 8 bit

  • the pixel data is normalized as values 0-255

This proposal consists of three key components to provide all of this information:

Type Denotation Definition#

To begin with, we define a set of semantic types that define what models generally consume as inputs and produce as outputs.

Specifically, in our first proposal we define the following set of standard denotations:

  1. TENSOR describes that a type holds a generic tensor using the standard TypeProto message.

  2. IMAGE describes that a type holds an image. You can use dimension denotation to learn more about the layout of the image, and also the optional model metadata_props.

  3. AUDIO describes that a type holds an audio clip.

  4. TEXT describes that a type holds a block of text.

Model authors SHOULD add type denotation to inputs and outputs for the model as appropriate.

An Example with input IMAGE#

Let’s use the same SqueezeNet example from above and show everything to properly annotate the model:

  • First set the TypeProto.denotation =IMAGE for the ValueInfoProto data_0

  • Because it’s an image, the model consumer now knows to go look for image metadata on the model

  • Then include 3 metadata strings on ModelProto.metadata_props

    • Image.BitmapPixelFormat = Bgr8

    • Image.ColorSpaceGamma = SRGB

    • Image.NominalPixelRange = NominalRange_0_255

  • For that same ValueInfoProto, make sure to also use Dimension Denotations to denote NCHW

    • TensorShapeProto.Dimension[0].denotation = DATA_BATCH

    • TensorShapeProto.Dimension[1].denotation = DATA_CHANNEL

    • TensorShapeProto.Dimension[2].denotation = DATA_FEATURE

    • TensorShapeProto.Dimension[3].denotation = DATA_FEATURE

Now there is enough information in the model to know everything about how to pass a correct image into the model.