Object Detection Concepts

Image classification predicts for a whole image. Object detection predicts a variable-size set of labeled boxes and scores, so evaluation must decide which prediction matches which target.

Prerequisites

Complete Vision Evaluation and Class Imbalance. Know precision, recall, and (x_min, y_min, x_max, y_max) coordinates.

Learning goals

• read the per-image target and prediction structures used by maintained detectors.
• calculate intersection over union (IoU) for valid boxes.
• distinguish classification, localization, duplicate, and missed-object errors.

Worked example

The prediction set contains boxes, labels, and confidence scores. Matching proceeds by score and requires both the correct class and IoU at least 0.5; one target can be matched once.

import torch

target_boxes = torch.tensor([[0., 0., 2., 2.], [4., 4., 6., 6.]])
target_labels = torch.tensor([1, 2])
predicted_boxes = torch.tensor([[0., 0., 2., 2.], [3.5, 3.5, 6., 6.], [0., 0., 1., 1.]])
predicted_labels = torch.tensor([1, 2, 2])
scores = torch.tensor([0.9, 0.8, 0.7])

def iou(first, second):
    top_left = torch.maximum(first[:2], second[:2])
    bottom_right = torch.minimum(first[2:], second[2:])
    intersection = (bottom_right - top_left).clamp(min=0).prod()
    first_area = (first[2:] - first[:2]).prod()
    second_area = (second[2:] - second[:2]).prod()
    return intersection / (first_area + second_area - intersection)

matched_targets = set()
true_positives = 0
for box, label, _ in zip(predicted_boxes, predicted_labels, scores):
    candidates = [(index, iou(box, target)) for index, target in enumerate(target_boxes)
                  if target_labels[index] == label and index not in matched_targets]
    if candidates:
        index, overlap = max(candidates, key=lambda item: float(item[1]))
        if overlap >= 0.5:
            matched_targets.add(index)
            true_positives += 1
false_positives = len(predicted_boxes) - true_positives
false_negatives = len(target_boxes) - true_positives

print(f"targets boxes={tuple(target_boxes.shape)} labels={tuple(target_labels.shape)}")
print(f"predictions boxes={tuple(predicted_boxes.shape)} labels={tuple(predicted_labels.shape)} scores={tuple(scores.shape)}")
print(f"matched_ious={[round(float(iou(predicted_boxes[i], target_boxes[i])), 2) for i in range(2)]}")
print(f"precision={true_positives / (true_positives + false_positives):.3f} recall={true_positives / (true_positives + false_negatives):.3f}")

Output:

targets boxes=(2, 4) labels=(2,)
predictions boxes=(3, 4) labels=(3,) scores=(3,)
matched_ious=[1.0, 0.64]
precision=0.667 recall=1.000

Production benchmarks usually aggregate precision and recall over score thresholds, classes, object sizes, and multiple IoU thresholds. Non-maximum suppression can remove duplicate boxes but can also suppress nearby real objects. This tiny matcher teaches the target/output contract; it is not a production detector or a complete average-precision implementation.

Common mistakes

• Mixing normalized coordinates with pixel coordinates or allowing negative-area boxes.
• Matching by IoU while ignoring class labels, or matching multiple predictions to one target.
• Calling one IoU threshold “the” detector quality.
• Evaluating only large, well-lit objects and missing size, occlusion, and slice failures.
• Treating confidence scores as calibrated probabilities.

Practice

Move the second predicted box to [5, 5, 7, 7] and recompute IoU. Explain how the result can be a localization false positive and a missed target simultaneously.

Next steps

Continue with Semantic Segmentation Concepts to move from boxes to one class decision per pixel.

Primary sources

Checked 2026-10-16: TorchVision object-detection finetuning tutorial, TorchVision detection model output contract, COCO evaluation API, and Everingham et al., the PASCAL Visual Object Classes challenge.