Skip to main content

Vision Evaluation and Class Imbalance

Overall accuracy weights each example equally, so a frequent class can hide failure on a rare but important class. Evaluation must preserve class support, decision costs, and an inspectable link back to examples.

Prerequisites​

Complete Transfer Learning and understand confusion matrices, precision, recall, and held-out evaluation.

Learning goals​

  • derive per-class recall and macro recall from a confusion matrix.
  • separate class imbalance remedies used during training from fixed evaluation reporting.
  • inspect concrete errors without turning the test set into a tuning set.

Worked example​

Rows below are true classes and columns are predicted classes. Class 2 appears once and is never recovered, although the majority class performs much better.

import torch

targets = torch.tensor([0, 0, 0, 0, 0, 1, 1, 2])
predictions = torch.tensor([0, 0, 0, 0, 1, 1, 0, 0])
class_names = ["circle", "square", "triangle"]
confusion = torch.zeros((3, 3), dtype=torch.int64)
for truth, predicted in zip(targets, predictions):
    confusion[truth, predicted] += 1
support = confusion.sum(dim=1)
recall = confusion.diag() / support
accuracy = confusion.diag().sum() / confusion.sum()
errors = [index for index, (truth, predicted) in enumerate(zip(targets, predictions)) if truth != predicted]

print(f"confusion={confusion.tolist()} support={support.tolist()}")
print(f"accuracy={accuracy:.3f} macro_recall={recall.mean():.3f}")
print(f"per_class_recall={dict(zip(class_names, [round(value, 3) for value in recall.tolist()]))}")
print(f"error_indices={errors}")

Output:

confusion=[[4, 1, 0], [1, 1, 0], [1, 0, 0]] support=[5, 2, 1]
accuracy=0.625 macro_recall=0.433
per_class_recall={'circle': 0.8, 'square': 0.5, 'triangle': 0.0}
error_indices=[4, 6, 7]

Use training-only strategies—class-weighted loss, balanced sampling, or defensible augmentation—to change learning. Report metrics on the natural validation/test distribution unless the deployment question explicitly defines another target distribution. Precision-recall curves, calibration, and cost-specific thresholds answer different questions; choose them before reading final test outcomes.

Inspect errors by stable sample ID or filename, true label, predicted label, score, and relevant slice. Repeatedly redesigning from test errors leaks test evidence into development. After such iteration, obtain a fresh untouched evaluation set.

Common mistakes​

  • Reporting accuracy without class supports or a confusion matrix.
  • Oversampling validation/test data and then presenting the result as natural prevalence.
  • Computing class weights from the full dataset rather than training labels only.
  • Choosing a threshold on the test set or equating a high score with calibrated probability.
  • Treating a tiny rare-class support as a precise recall estimate.

Practice​

Compute class-2 precision and explain why it differs from recall. Then assign a higher cost to missing triangles and state which validation decision—not test decision—should change.

Next steps​

Continue with Object Detection Concepts, where matching and localization add new error types.

Primary sources​

Checked 2026-10-16: PyTorch CrossEntropyLoss class weights, PyTorch WeightedRandomSampler, scikit-learn confusion matrix, and scikit-learn balanced accuracy.