Skip to main content

Vision Embeddings and Similarity

A vision embedding maps an image to a numeric vector. Nearby vectors are useful only if the encoder, preprocessing, similarity rule, and validation evidence make “nearby” meaningful for the task.

Prerequisites​

Complete Transfer Learning and be comfortable with dot products and vector norms.

Learning goals​

  • distinguish classifier logits from intermediate embeddings.
  • L2-normalize vectors before using a dot product as cosine similarity.
  • evaluate retrieval with labels and hard errors rather than attractive nearest-neighbor examples alone.

Worked example​

Four local, hand-auditable embeddings represent a query, a nearby item, an orthogonal item, and an opposite direction. Normalization makes the dot product equal cosine similarity.

import torch
from torch.nn import functional as F

names = ["query", "nearby", "orthogonal", "opposite"]
embeddings = torch.tensor([[1.0, 0.0], [0.8, 0.6], [0.0, 1.0], [-1.0, 0.0]])
normalized = F.normalize(embeddings, p=2, dim=1)
similarities = normalized @ normalized.T
query_scores = similarities[0].clone()
query_scores[0] = -float("inf")
neighbor = int(query_scores.argmax())

print(f"embeddings={tuple(normalized.shape)} norms={[round(v, 1) for v in normalized.norm(dim=1).tolist()]}")
print(f"query_similarities={[round(v, 1) for v in similarities[0].tolist()]}")
print(f"nearest={names[neighbor]} score={similarities[0, neighbor]:.1f}")

Output:

embeddings=(4, 2) norms=[1.0, 1.0, 1.0, 1.0]
query_similarities=[1.0, 0.8, 0.0, -1.0]
nearest=nearby score=0.8

For a CNN, an embedding often comes from the pooled backbone before the classifier head. It remains versioned model output: changing weights, preprocessing, or dimensionality invalidates a stored index unless compatibility is established. Retrieval quality needs task labels or judgments, metrics such as recall at kk, slice analysis, and inspection of false neighbors.

Common mistakes​

  • Comparing unnormalized vectors with a dot product and calling it cosine similarity.
  • Mixing embeddings produced by different model or preprocessing versions.
  • Using classifier logits as a universal semantic representation without validation.
  • Selecting a similarity threshold from final test examples.
  • Assuming visual proximity is fair, causal, private, or safe for identity-sensitive use.

Practice​

Add [0.6, 0.8] and rank the non-query items. Then define what a relevant neighbor means for one concrete application before choosing recall at kk.

Next steps​

Finish the batch with Project: Image Classifier with Local Fixtures, which joins preprocessing, splitting, training, metrics, and error inspection.

Primary sources​

Checked 2026-10-16: PyTorch normalize, TorchVision feature extraction, Kornblith et al., similarity of neural network representations, and Radford et al., learning transferable visual models.