Skip to main content

Top-K Ranking Metrics

Top-k metrics evaluate an ordered prefix, not a model in isolation. A result is interpretable only with its relevance definition, cutoff, candidate set, denominator, user inclusion rule, and aggregation method.

Prerequisites​

You should have a frozen split and the baseline rankings produced from identical candidates.

Learning goals​

  • Compute Precision@k, Recall@k, MRR@k, and NDCG@k by hand.
  • State each metric's numerator and denominator.
  • Define behavior for empty relevance sets, short lists, and duplicate recommendations.
  • Explain why an offline metric is not product impact.

Metric contract​

For this worked case, relevance means a binary positive test action, the candidate set is the five eligible unseen items from the previous lesson, the cutoff is k = 3, and the relevant set is {i3, i5}.

  • Precision@k divides relevant unique recommendations in the first k slots by k. This fixed denominator penalizes short lists.
  • Recall@k divides those hits by the number of relevant candidate items. For an empty relevant set, this lesson returns 0 and reports the cohort separately.
  • MRR@k is the reciprocal rank of the first relevant item, or 0 when none appears.
  • NDCG@k divides discounted cumulative gain by the ideal DCG for the same relevance judgments and cutoff. This example uses binary gain and log2(rank + 1) discount.

Duplicate recommendations are invalid and rejected rather than counted twice. Production protocols should also define how to handle unavailable items and fewer than k eligible candidates.

Worked example: hand-checkable metrics​

import math

def checked_prefix(recommendations, k):
prefix = recommendations[:k]
if len(prefix) != len(set(prefix)):
raise ValueError("duplicate recommendations")
return prefix

def metrics_at_k(recommendations, relevant, k):
prefix = checked_prefix(recommendations, k)
hits = [item in relevant for item in prefix]
precision = sum(hits) / k if k else 0.0
recall = sum(hits) / len(relevant) if relevant else 0.0
first = next((rank for rank, hit in enumerate(hits, start=1) if hit), None)
mrr = 1 / first if first else 0.0
dcg = sum(hit / math.log2(rank + 1) for rank, hit in enumerate(hits, start=1))
ideal_hits = [1] * min(len(relevant), k)
idcg = sum(hit / math.log2(rank + 1) for rank, hit in enumerate(ideal_hits, start=1))
ndcg = dcg / idcg if idcg else 0.0
return precision, recall, mrr, ndcg

values = metrics_at_k(["i5", "i4", "i3"], {"i3", "i5"}, 3)
print("precision@3={:.3f} recall@3={:.3f} mrr@3={:.3f} ndcg@3={:.3f}".format(*values))
empty = metrics_at_k(["i5"], set(), 3)[1]
try:
metrics_at_k(["i5", "i5"], {"i5"}, 2)
rejected = False
except ValueError:
rejected = True
print(f"empty={empty:.3f} duplicates_rejected={rejected}")

Output:

precision@3=0.667 recall@3=1.000 mrr@3=1.000 ndcg@3=0.920
empty=0.000 duplicates_rejected=True

Aggregate without hiding the population​

Macro averaging gives each included user equal weight; micro aggregation weights users by the counts contributing to the numerator and denominator. They answer different questions. Report the number of evaluated users, cold-start cohorts, empty-relevance users, and uncertainty across users or time windows.

Candidate sampling also changes the task. Ranking against 99 sampled negatives is easier than ranking against a full eligible catalog and cannot be compared without the sampling protocol. Metrics based only on logged positives additionally inherit exposure and selection bias.

Limitations and common mistakes​

  • Precision can reward conservative lists while recall rewards finding more of a finite relevant set.
  • MRR ignores every hit after the first.
  • NDCG depends on the gain and discount definitions; variants with the same name can differ.
  • Empty-relevance users must not disappear silently from the denominator.
  • Offline relevance measures logged behavior under historical exposure, not satisfaction or causal lift.

Practice​

Recompute all four metrics for [i4, i3, i5]. Then remove i5 from the candidate set and explain why comparing the two scores would confound ranking quality with candidate construction.

Next steps​

Accuracy is only one dimension. Next, measure diversity, coverage, novelty, and serendipity.

Primary sources​

Checked 2026-10-17: Järvelin and Kekäläinen, “Cumulated Gain-Based Evaluation of IR Techniques”; Herlocker et al., “Evaluating Collaborative Filtering Recommender Systems”; and Schnabel et al., “Recommendations as Treatments”.