NegativeLogLikelihoodLoss

NegativeLogLikelihoodLoss - 13

Version

• name: NegativeLogLikelihoodLoss (GitHub)

• domain: main

• since_version: 13

• function: False

• support_level: SupportType.COMMON

• shape inference: True

This version of the operator has been available since version 13.

Summary

A NegativeLogLikelihoodLoss operator computes (weighted) negative log likelihood loss. Its “input” tensor has the shape of (N, C, d1, d2, …, dk) where k >= 0. The “input” tensor contains log-probabilities for input[n, :, d_1, d_2,…, d_k] being in a class of [0, C). The operator’s “target” input tensor has the shape of (N, d1, d2, …, dk). It encodes class labels (one of C classes) or it may contain a special value (indicated by an attribute ignore_index) for N x d1 x d2 x … x dk samples. The loss value for input[n, :, d_1, d_2,…d_k] being classified as class c = target[n][d_1][d_2]…[d_k] is computed as:

loss[n][d_1][d_2]…[d_k] = -input[n][c][d_1][d_2]…[d_k].

When an optional “weight” is provided, the sample loss is calculated as:

loss[n][d_1][d_2]…[d_k] = -input[n][c][d_1][d_2]…[d_k] * weight[c].

loss is zero for the case when target-value equals ignore_index.

loss[n][d_1][d_2]…[d_k] = 0, when target[n][d_1][d_2]…[d_k] = ignore_index

If “reduction” attribute is set to “none”, the operator’s output will be the above loss with shape (N, d1, d2, …, dk). If “reduction” attribute is set to “mean” (the default attribute value), the output loss is (weight) averaged:

mean(loss), if “weight” is not provided,

or if weight is provided,

sum(loss) / sum(weight[target[n][d_1][d_2]…[d_k]]]), for all samples.

If “reduction” attribute is set to “sum”, the output is a scalar:

sum(loss).

See also https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss.

Example 1:

// negative log likelihood loss, “none” reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]]

loss = np.zeros((N, d1)) for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1]

// print(loss) // [[-3. -2.] // [-0. -2.]]

Example 2:

// weighted negative log likelihood loss, sum reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]] weight = [0.2, 0.3, 0.1] loss = np.zeros((N, d1)) for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1] * weight[c]

loss = np.sum(loss) // print(loss) // -1.1

Example 3:

// weighted negative log likelihood loss, mean reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]] weight = [0.2, 0.3, 0.1] loss = np.zeros((N, d1)) weight_total = 0 for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1] * weight[c] weight_total = weight_total + weight[c]

loss = np.sum(loss) / weight_total // print(loss) // -1.57

Attributes

• ignore_index: Specifies a target value that is ignored and does not contribute to the input gradient. It’s an optional value.

• reduction: Type of reduction to apply to loss: none, sum, mean (default). ‘none’: the output is the loss for each sample. ‘sum’: the output will be summed. ‘mean’: the sum of the output will be divided by the sum of applied weights. Default value is 'mean'.

Inputs

Between 2 and 3 inputs.

• input (heterogeneous) - T: Input tensor of shape (N, C) or (N, C, d1, d2, …, dk).

• target (heterogeneous) - Tind: Target tensor of shape (N) or (N, d1, d2, …, dk). Target element value shall be in range of [0, C). If ignore_index is specified, it may have a value outside [0, C) and the target values should either be in the range [0, C) or have the value ignore_index.

• weight (optional, heterogeneous) - T: Optional rescaling weight tensor. If given, it has to be a tensor of size C. Otherwise, it is treated as if having all ones.

Outputs

• loss (heterogeneous) - T: The negative log likelihood loss

Type Constraints

• T in ( tensor(double), tensor(float), tensor(float16) ): Constrain input, weight, and output types to floating-point tensors.

• Tind in ( tensor(int32), tensor(int64) ): Constrain target to integer types

Examples

_input_shape_is_NC

reduction = "none"
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target"],
    outputs=["loss"],
    reduction=reduction,
)

N, C = 3, 5
np.random.seed(0)
input = np.random.rand(N, C).astype(np.float32)
target = np.random.randint(0, high=C, size=(N,)).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=None, reduction=reduction
)

expect(
    node,
    inputs=[input, target],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NC",
)

_input_shape_is_NCd1d2

reduction = "none"
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target"],
    outputs=["loss"],
    reduction=reduction,
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=None, reduction=reduction
)

expect(
    node,
    inputs=[input, target],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2",
)

_input_shape_is_NCd1d2_reduction_mean

reduction = "mean"
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target"],
    outputs=["loss"],
    reduction=reduction,
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=None, reduction=reduction
)

expect(
    node,
    inputs=[input, target],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2_reduction_mean",
)

_input_shape_is_NCd1d2_reduction_sum

reduction = "sum"
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target"],
    outputs=["loss"],
    reduction=reduction,
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=None, reduction=reduction
)

expect(
    node,
    inputs=[input, target],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2_reduction_sum",
)

_input_shape_is_NCd1d2_with_weight

reduction = "none"
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target", "weight"],
    outputs=["loss"],
    reduction=reduction,
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=weight, reduction=reduction
)

expect(
    node,
    inputs=[input, target, weight],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2_with_weight",
)

_input_shape_is_NCd1d2_with_weight_reduction_mean

reduction = "mean"
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target", "weight"],
    outputs=["loss"],
    reduction=reduction,
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=weight, reduction=reduction
)

expect(
    node,
    inputs=[input, target, weight],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2_with_weight_reduction_mean",
)

_input_shape_is_NCd1d2_with_weight_reduction_sum

reduction = "sum"
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target", "weight"],
    outputs=["loss"],
    reduction=reduction,
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=weight, reduction=reduction
)

expect(
    node,
    inputs=[input, target, weight],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2_with_weight_reduction_sum",
)

_input_shape_is_NCd1d2_with_weight_reduction_sum_ii

reduction = "sum"
ignore_index = np.int64(0)
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target", "weight"],
    outputs=["loss"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)
target[0][0][0] = np.int64(0)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=weight, reduction=reduction, ignore_index=ignore_index
)

expect(
    node,
    inputs=[input, target, weight],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2_with_weight_reduction_sum_ii",
)

_input_shape_is_NCd1d2_no_weight_reduction_mean_ii

reduction = "mean"
ignore_index = np.int64(1)
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target"],
    outputs=["loss"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)
target[0][0][0] = np.int64(1)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, reduction=reduction, ignore_index=ignore_index
)

expect(
    node,
    inputs=[input, target],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2_no_weight_reduction_mean_ii",
)

_input_shape_is_NCd1

reduction = "mean"
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target"],
    outputs=["loss"],
    reduction=reduction,
)

N, C, d1 = 3, 5, 2
np.random.seed(0)
input = np.random.rand(N, C, d1).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, d1)).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=None, reduction=reduction
)

expect(
    node,
    inputs=[input, target],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1",
)

_input_shape_is_NCd1_weight

reduction = "mean"
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target", "weight"],
    outputs=["loss"],
    reduction=reduction,
)

N, C, d1 = 3, 5, 2
np.random.seed(0)
input = np.random.rand(N, C, d1).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, d1)).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=weight, reduction=reduction
)

expect(
    node,
    inputs=[input, target, weight],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1_weight",
)

_input_shape_is_NCd1_ii

reduction = "mean"
ignore_index = np.int64(1)
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target"],
    outputs=["loss"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C, d1 = 3, 5, 2
np.random.seed(0)
input = np.random.rand(N, C, d1).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, d1)).astype(np.int64)
target[0][0] = np.int64(1)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=None, reduction=reduction, ignore_index=ignore_index
)

expect(
    node,
    inputs=[input, target],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1_ii",
)

_input_shape_is_NCd1_weight_ii

reduction = "mean"
ignore_index = np.int64(1)
node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target", "weight"],
    outputs=["loss"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C, d1 = 3, 5, 2
np.random.seed(0)
input = np.random.rand(N, C, d1).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, d1)).astype(np.int64)
target[0][0] = np.int64(1)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=weight, reduction=reduction, ignore_index=ignore_index
)

expect(
    node,
    inputs=[input, target, weight],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1_weight_ii",
)

_input_shape_is_NCd1d2d3d4d5_mean_weight

reduction = "mean"

node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target", "weight"],
    outputs=["loss"],
    reduction=reduction,
)

N, C, dim1, dim2, dim3, dim4, dim5 = 3, 5, 6, 6, 5, 3, 4
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2, dim3, dim4, dim5).astype(np.float32)
target = np.random.randint(
    0, high=C, size=(N, dim1, dim2, dim3, dim4, dim5)
).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=weight, reduction=reduction
)

expect(
    node,
    inputs=[input, target, weight],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2d3d4d5_mean_weight",
)

_input_shape_is_NCd1d2d3d4d5_none_no_weight

reduction = "none"

node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target"],
    outputs=["loss"],
    reduction=reduction,
)

N, C, dim1, dim2, dim3, dim4, dim5 = 3, 5, 6, 6, 5, 3, 4
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2, dim3, dim4, dim5).astype(np.float32)
target = np.random.randint(
    0, high=C, size=(N, dim1, dim2, dim3, dim4, dim5)
).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, reduction=reduction
)

expect(
    node,
    inputs=[input, target],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2d3d4d5_none_no_weight",
)

_input_shape_is_NCd1_mean_weight_negative_ii

reduction = "mean"
ignore_index = np.int64(-1)

node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target", "weight"],
    outputs=["loss"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C, dim1 = 3, 5, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1)).astype(np.int64)
target[0][0] = -1
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=weight, reduction=reduction, ignore_index=ignore_index
)

expect(
    node,
    inputs=[input, target, weight],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1_mean_weight_negative_ii",
)

_input_shape_is_NCd1d2d3_none_no_weight_negative_ii

reduction = "none"
ignore_index = np.int64(-5)

node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target"],
    outputs=["loss"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C, dim1, dim2, dim3 = 3, 5, 6, 6, 5
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2, dim3).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2, dim3)).astype(
    np.int64
)
target[0][0][0][0] = -5

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, reduction=reduction, ignore_index=ignore_index
)

expect(
    node,
    inputs=[input, target],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2d3_none_no_weight_negative_ii",
)

_input_shape_is_NCd1d2d3_sum_weight_high_ii

reduction = "sum"
ignore_index = np.int64(10)

node = onnx.helper.make_node(
    "NegativeLogLikelihoodLoss",
    inputs=["input", "target", "weight"],
    outputs=["loss"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C = 3, 5
np.random.seed(0)
input = np.random.rand(N, C).astype(np.float32)
target = np.random.randint(0, high=C, size=(N)).astype(np.int64)
target[0] = 10
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(
    input, target, weight=weight, reduction=reduction, ignore_index=ignore_index
)

expect(
    node,
    inputs=[input, target, weight],
    outputs=[negative_log_likelihood_loss],
    name="test_nllloss_NCd1d2d3_sum_weight_high_ii",
)

Differences

0	0	A NegativeLogLikelihoodLoss operator computes (weighted) negative log likelihood loss.	A NegativeLogLikelihoodLoss operator computes (weighted) negative log likelihood loss.
1	1	Its "input" tensor has the shape of (N, C, d1, d2, ..., dk) where k >= 0.	Its "input" tensor has the shape of (N, C, d1, d2, ..., dk) where k >= 0.
2	2	The "input" tensor contains log-probabilities for input[n, :, d_1, d_2,..., d_k] being in a class of [0, C).	The "input" tensor contains log-probabilities for input[n, :, d_1, d_2,..., d_k] being in a class of [0, C).
3	3	The operator's "target" input tensor has the shape of (N, d1, d2, ..., dk). It encodes class labels (one of C classes)	The operator's "target" input tensor has the shape of (N, d1, d2, ..., dk). It encodes class labels (one of C classes)
4	4	or it may contain a special value (indicated by an attribute ignore_index) for N x d1 x d2 x ... x dk samples.	or it may contain a special value (indicated by an attribute ignore_index) for N x d1 x d2 x ... x dk samples.
5	5	The loss value for input[n, :, d_1, d_2,...d_k] being classified as class c = target[n][d_1][d_2]...[d_k] is computed as:	The loss value for input[n, :, d_1, d_2,...d_k] being classified as class c = target[n][d_1][d_2]...[d_k] is computed as:
	6
6	7	loss[n][d_1][d_2]...[d_k] = -input[n][c][d_1][d_2]...[d_k].	loss[n][d_1][d_2]...[d_k] = -input[n][c][d_1][d_2]...[d_k].
	8
7	9	When an optional "weight" is provided, the sample loss is calculated as:	When an optional "weight" is provided, the sample loss is calculated as:
	10
8	11	loss[n][d_1][d_2]...[d_k] = -input[n][c][d_1][d_2]...[d_k] * weight[c].	loss[n][d_1][d_2]...[d_k] = -input[n][c][d_1][d_2]...[d_k] * weight[c].
	12
9	13	loss is zero for the case when target-value equals ignore_index.	loss is zero for the case when target-value equals ignore_index.
10	14
11	15	loss[n][d_1][d_2]...[d_k] = 0, when target[n][d_1][d_2]...[d_k] = ignore_index	loss[n][d_1][d_2]...[d_k] = 0, when target[n][d_1][d_2]...[d_k] = ignore_index
	16
12	17	If "reduction" attribute is set to "none", the operator's output will be the above loss with shape (N, d1, d2, ..., dk).	If "reduction" attribute is set to "none", the operator's output will be the above loss with shape (N, d1, d2, ..., dk).
13	18	If "reduction" attribute is set to "mean" (the default attribute value), the output loss is (weight) averaged:	If "reduction" attribute is set to "mean" (the default attribute value), the output loss is (weight) averaged:
	19
14	20	mean(loss), if "weight" is not provided,	mean(loss), if "weight" is not provided,
	21
15	22	or if weight is provided,	or if weight is provided,
	23
16	24	sum(loss) / sum(weight[target[n][d_1][d_2]...[d_k]]]), for all samples.	sum(loss) / sum(weight[target[n][d_1][d_2]...[d_k]]]), for all samples.
	25
17	26	If "reduction" attribute is set to "sum", the output is a scalar:	If "reduction" attribute is set to "sum", the output is a scalar:
18	27	sum(loss).	sum(loss).
	28
19	29	See also https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss.	See also https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss.
	30
20	31	Example 1:	Example 1:
	32
21	33	// negative log likelihood loss, "none" reduction	// negative log likelihood loss, "none" reduction
22	34	N, C, d1 = 2, 3, 2	N, C, d1 = 2, 3, 2
23	35	input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],	input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],
24	36	[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]	[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]
25	37	target = [[2, 1], [0, 2]]	target = [[2, 1], [0, 2]]
	38
26	39	loss = np.zeros((N, d1))	loss = np.zeros((N, d1))
27	40	for n in range(N):	for n in range(N):
28	41	for d_1 in range(d1):	for d_1 in range(d1):
29	42	c = target[n][d_1]	c = target[n][d_1]
30	43	loss[n][d_1] = -input[n][c][d_1]	loss[n][d_1] = -input[n][c][d_1]
	44
31	45	// print(loss)	// print(loss)
32	46	// [[-3. -2.]	// [[-3. -2.]
33	47	// [-0. -2.]]	// [-0. -2.]]
	48
34	49	Example 2:	Example 2:
	50
35	51	// weighted negative log likelihood loss, sum reduction	// weighted negative log likelihood loss, sum reduction
36	52	N, C, d1 = 2, 3, 2	N, C, d1 = 2, 3, 2
37	53	input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],	input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],
38	54	[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]	[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]
39	55	target = [[2, 1], [0, 2]]	target = [[2, 1], [0, 2]]
40	56	weight = [0.2, 0.3, 0.1]	weight = [0.2, 0.3, 0.1]
41	57	loss = np.zeros((N, d1))	loss = np.zeros((N, d1))
42	58	for n in range(N):	for n in range(N):
43	59	for d_1 in range(d1):	for d_1 in range(d1):
44	60	c = target[n][d_1]	c = target[n][d_1]
45	61	loss[n][d_1] = -input[n][c][d_1] * weight[c]	loss[n][d_1] = -input[n][c][d_1] * weight[c]
	62
46	63	loss = np.sum(loss)	loss = np.sum(loss)
47	64	// print(loss)	// print(loss)
48	65	// -1.1	// -1.1
	66
49	67	Example 3:	Example 3:
	68
50	69	// weighted negative log likelihood loss, mean reduction	// weighted negative log likelihood loss, mean reduction
51	70	N, C, d1 = 2, 3, 2	N, C, d1 = 2, 3, 2
52	71	input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],	input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],
53	72	[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]	[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]
54	73	target = [[2, 1], [0, 2]]	target = [[2, 1], [0, 2]]
55	74	weight = [0.2, 0.3, 0.1]	weight = [0.2, 0.3, 0.1]
56	75	loss = np.zeros((N, d1))	loss = np.zeros((N, d1))
57	76	weight_total = 0	weight_total = 0
58	77	for n in range(N):	for n in range(N):
59	78	for d_1 in range(d1):	for d_1 in range(d1):
60	79	c = target[n][d_1]	c = target[n][d_1]
61	80	loss[n][d_1] = -input[n][c][d_1] * weight[c]	loss[n][d_1] = -input[n][c][d_1] * weight[c]
62	81	weight_total = weight_total + weight[c]	weight_total = weight_total + weight[c]
	82
63	83	loss = np.sum(loss) / weight_total	loss = np.sum(loss) / weight_total
64	84	// print(loss)	// print(loss)
65	85	// -1.57	// -1.57
66	86
67	87	**Attributes**	**Attributes**
68	88
69	89	* **ignore_index**:	* **ignore_index**:
70	90	Specifies a target value that is ignored and does not contribute to	Specifies a target value that is ignored and does not contribute to
71	91	the input gradient. It's an optional value.	the input gradient. It's an optional value.
72	92	* **reduction**:	* **reduction**:
73	93	Type of reduction to apply to loss: none, sum, mean (default).	Type of reduction to apply to loss: none, sum, mean (default).
74	94	'none': the output is the loss for each sample. 'sum': the output	'none': the output is the loss for each sample. 'sum': the output
75	95	will be summed. 'mean': the sum of the output will be divided by the	will be summed. 'mean': the sum of the output will be divided by the
76	96	sum of applied weights. Default value is 'mean'.	sum of applied weights. Default value is 'mean'.
77	97
78	98	**Inputs**	**Inputs**
79	99
80	100	Between 2 and 3 inputs.	Between 2 and 3 inputs.
81	101
82	102	* **input** (heterogeneous) - **T**:	* **input** (heterogeneous) - **T**:
83	103	Input tensor of shape (N, C) or (N, C, d1, d2, ..., dk).	Input tensor of shape (N, C) or (N, C, d1, d2, ..., dk).
84	104	* **target** (heterogeneous) - **Tind**:	* **target** (heterogeneous) - **Tind**:
85	105	Target tensor of shape (N) or (N, d1, d2, ..., dk). Target element	Target tensor of shape (N) or (N, d1, d2, ..., dk). Target element
86	106	value shall be in range of [0, C). If ignore_index is specified, it	value shall be in range of [0, C). If ignore_index is specified, it
87	107	may have a value outside [0, C) and the target values should either	may have a value outside [0, C) and the target values should either
88	108	be in the range [0, C) or have the value ignore_index.	be in the range [0, C) or have the value ignore_index.
89	109	* **weight** (optional, heterogeneous) - **T**:	* **weight** (optional, heterogeneous) - **T**:
90	110	Optional rescaling weight tensor. If given, it has to be a tensor of	Optional rescaling weight tensor. If given, it has to be a tensor of
91	111	size C. Otherwise, it is treated as if having all ones.	size C. Otherwise, it is treated as if having all ones.
92	112
93	113	**Outputs**	**Outputs**
94	114
95	115	* **loss** (heterogeneous) - **T**:	* **loss** (heterogeneous) - **T**:
96	116	The negative log likelihood loss	The negative log likelihood loss
97	117
98	118	**Type Constraints**	**Type Constraints**
99	119
100	120	* **T** in (	* **T** in (
101	121	tensor(double),	tensor(double),
102	122	tensor(float),	tensor(float),
103	123	tensor(float16)	tensor(float16)
104	124	):	):
105	125	Constrain input, weight, and output types to floating-point tensors.	Constrain input, weight, and output types to floating-point tensors.
106	126	* **Tind** in (	* **Tind** in (
107	127	tensor(int32),	tensor(int32),
108	128	tensor(int64)	tensor(int64)
109	129	):	):
110	130	Constrain target to integer types	Constrain target to integer types

NegativeLogLikelihoodLoss - 12

Version

• name: NegativeLogLikelihoodLoss (GitHub)

• domain: main

• since_version: 12

• function: False

• support_level: SupportType.COMMON

• shape inference: True

This version of the operator has been available since version 12.

Summary

A NegativeLogLikelihoodLoss operator computes (weighted) negative log likelihood loss. Its “input” tensor has the shape of (N, C, d1, d2, …, dk) where k >= 0. The “input” tensor contains log-probabilities for input[n, :, d_1, d_2,…, d_k] being in a class of [0, C). The operator’s “target” input tensor has the shape of (N, d1, d2, …, dk). It encodes class labels (one of C classes) or it may contain a special value (indicated by an attribute ignore_index) for N x d1 x d2 x … x dk samples. The loss value for input[n, :, d_1, d_2,…d_k] being classified as class c = target[n][d_1][d_2]…[d_k] is computed as:

loss[n][d_1][d_2]…[d_k] = -input[n][c][d_1][d_2]…[d_k].

When an optional “weight” is provided, the sample loss is calculated as:

loss[n][d_1][d_2]…[d_k] = -input[n][c][d_1][d_2]…[d_k] * weight[c].

loss is zero for the case when target-value equals ignore_index.

loss[n][d_1][d_2]…[d_k] = 0, when target[n][d_1][d_2]…[d_k] = ignore_index

If “reduction” attribute is set to “none”, the operator’s output will be the above loss with shape (N, d1, d2, …, dk). If “reduction” attribute is set to “mean” (the default attribute value), the output loss is (weight) averaged:

mean(loss), if “weight” is not provided,

or if weight is provided,

sum(loss) / sum(weight[target[n][d_1][d_2]…[d_k]]]), for all samples.

If “reduction” attribute is set to “sum”, the output is a scalar:

sum(loss).

See also https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss. Example 1:

// negative log likelihood loss, “none” reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]] loss = np.zeros((N, d1)) for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1]

// print(loss) // [[-3. -2.] // [-0. -2.]]

Example 2:

// weighted negative log likelihood loss, sum reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]] weight = [0.2, 0.3, 0.1] loss = np.zeros((N, d1)) for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1] * weight[c]

loss = np.sum(loss) // print(loss) // -1.1

Example 3:

// weighted negative log likelihood loss, mean reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]] weight = [0.2, 0.3, 0.1] loss = np.zeros((N, d1)) weight_total = 0 for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1] * weight[c] weight_total = weight_total + weight[c]

loss = np.sum(loss) / weight_total // print(loss) // -1.57

Attributes

• ignore_index: Specifies a target value that is ignored and does not contribute to the input gradient. It’s an optional value.

• reduction: Type of reduction to apply to loss: none, sum, mean (default). ‘none’: the output is the loss for each sample. ‘sum’: the output will be summed. ‘mean’: the sum of the output will be divided by the sum of applied weights. Default value is 'mean'.

Inputs

Between 2 and 3 inputs.

• input (heterogeneous) - T: Input tensor of shape (N, C) or (N, C, d1, d2, …, dk).

• target (heterogeneous) - Tind: Target tensor of shape (N) or (N, d1, d2, …, dk). Target element value shall be in range of [0, C). If ignore_index is specified, it may have a value outside [0, C) and the target values should either be in the range [0, C) or have the value ignore_index.

• weight (optional, heterogeneous) - T: Optional rescaling weight tensor. If given, it has to be a tensor of size C. Otherwise, it is treated as if having all ones.

Outputs

• loss (heterogeneous) - T: The negative log likelihood loss

Type Constraints

• T in ( tensor(double), tensor(float), tensor(float16) ): Constrain input, weight, and output types to floating-point tensors.

• Tind in ( tensor(int32), tensor(int64) ): Constrain target to integer types