Semantic Segmentation Concepts

Semantic segmentation assigns a class to every pixel. It does not distinguish two instances of the same class; instance and panoptic segmentation answer different questions.

Prerequisites

Complete Object Detection Concepts and understand image resizing and interpolation.

Learning goals

• align dense logits (N, classes, H, W) with integer targets (N, H, W).
• compute intersection over union per class and report the aggregation rule.
• protect discrete masks from interpolation and alignment errors.

Worked example

This two-class mask uses 0 for background and 1 for foreground. The predicted mask has one missed foreground pixel and one false foreground pixel.

import torch
from torch.nn import functional as F

target = torch.tensor([[[0, 0, 0, 0],
                        [0, 1, 1, 0],
                        [0, 1, 1, 0],
                        [0, 0, 0, 0]]])
predicted = torch.tensor([[[0, 0, 0, 0],
                           [0, 1, 1, 0],
                           [0, 1, 0, 0],
                           [0, 0, 1, 0]]])
logits = F.one_hot(predicted, num_classes=2).permute(0, 3, 1, 2).to(torch.float32) * 4
decoded = logits.argmax(dim=1)

ious = []
for class_id in range(2):
    truth = target == class_id
    guess = decoded == class_id
    intersection = (truth & guess).sum()
    union = (truth | guess).sum()
    ious.append(float(intersection / union))
pixel_accuracy = float((decoded == target).float().mean())

print(f"logits={tuple(logits.shape)} target={tuple(target.shape)} decoded={tuple(decoded.shape)}")
print(f"iou_background={ious[0]:.3f} iou_foreground={ious[1]:.3f} mean_iou={sum(ious) / 2:.3f}")
print(f"pixel_accuracy={pixel_accuracy:.3f}")

Output:

logits=(1, 2, 4, 4) target=(1, 4, 4) decoded=(1, 4, 4)
iou_background=0.846 iou_foreground=0.600 mean_iou=0.723
pixel_accuracy=0.875

Pixel accuracy is dominated by background in many datasets. Report per-class IoU, class support, the mean's included/excluded classes, and behavior for absent classes. Resize photographs with an image interpolation method but masks with nearest-neighbor interpolation. Apply the same crop and geometric transform to both.

Common mistakes

• Giving CrossEntropyLoss a one-hot target instead of integer class IDs for this interface.
• Resizing class masks with bilinear interpolation, creating invalid fractional labels.
• Comparing logits and targets at different spatial resolutions without an explicit alignment policy.
• Reporting background-heavy pixel accuracy alone.
• Treating semantic masks as object instances or ignoring annotation-boundary uncertainty.

Practice

Remove the false foreground pixel at row 3, column 2 and recompute both IoUs. Explain why foreground IoU changes more than pixel accuracy.

Next steps

Continue with Vision Embeddings and Similarity to compare images in a learned representation space.

Primary sources

Checked 2026-10-16: TorchVision semantic-segmentation model outputs, PyTorch CrossEntropyLoss, TorchVision Resize, and Long et al., fully convolutional networks.