Top-K Ranking Metrics
Top-k metrics evaluate an ordered prefix, not a model in isolation. A result is interpretable only with its relevance definition, cutoff, candidate set, denominator, user inclusion rule, and aggregation method.
Prerequisites
You should have a frozen split and the baseline rankings produced from identical candidates.
Learning goals
- Compute Precision@k, Recall@k, MRR@k, and NDCG@k by hand.
- State each metric's numerator and denominator.
- Define behavior for empty relevance sets, short lists, and duplicate recommendations.
- Explain why an offline metric is not product impact.
Metric contract
For this worked case, relevance means a binary positive test action, the candidate set is the five eligible unseen items from the previous lesson, the cutoff is k = 3, and the relevant set is {i3, i5}.
- Precision@k divides relevant unique recommendations in the first
kslots byk. This fixed denominator penalizes short lists. - Recall@k divides those hits by the number of relevant candidate items. For an empty relevant set, this lesson returns
0and reports the cohort separately. - MRR@k is the reciprocal rank of the first relevant item, or
0when none appears. - NDCG@k divides discounted cumulative gain by the ideal DCG for the same relevance judgments and cutoff. This example uses binary gain and
log2(rank + 1)discount.
Duplicate recommendations are invalid and rejected rather than counted twice. Production protocols should also define how to handle unavailable items and fewer than k eligible candidates.
Worked example: hand-checkable metrics
import math
def checked_prefix(recommendations, k):
prefix = recommendations[:k]
if len(prefix) != len(set(prefix)):
raise ValueError("duplicate recommendations")
return prefix
def metrics_at_k(recommendations, relevant, k):
prefix = checked_prefix(recommendations, k)
hits = [item in relevant for item in prefix]
precision = sum(hits) / k if k else 0.0
recall = sum(hits) / len(relevant) if relevant else 0.0
first = next((rank for rank, hit in enumerate(hits, start=1) if hit), None)
mrr = 1 / first if first else 0.0
dcg = sum(hit / math.log2(rank + 1) for rank, hit in enumerate(hits, start=1))
ideal_hits = [1] * min(len(relevant), k)
idcg = sum(hit / math.log2(rank + 1) for rank, hit in enumerate(ideal_hits, start=1))
ndcg = dcg / idcg if idcg else 0.0
return precision, recall, mrr, ndcg
values = metrics_at_k(["i5", "i4", "i3"], {"i3", "i5"}, 3)
print("precision@3={:.3f} recall@3={:.3f} mrr@3={:.3f} ndcg@3={:.3f}".format(*values))
empty = metrics_at_k(["i5"], set(), 3)[1]
try:
metrics_at_k(["i5", "i5"], {"i5"}, 2)
rejected = False
except ValueError:
rejected = True
print(f"empty={empty:.3f} duplicates_rejected={rejected}")
Output:
precision@3=0.667 recall@3=1.000 mrr@3=1.000 ndcg@3=0.920
empty=0.000 duplicates_rejected=True
Aggregate without hiding the population
Macro averaging gives each included user equal weight; micro aggregation weights users by the counts contributing to the numerator and denominator. They answer different questions. Report the number of evaluated users, cold-start cohorts, empty-relevance users, and uncertainty across users or time windows.
Candidate sampling also changes the task. Ranking against 99 sampled negatives is easier than ranking against a full eligible catalog and cannot be compared without the sampling protocol. Metrics based only on logged positives additionally inherit exposure and selection bias.
Limitations and common mistakes
- Precision can reward conservative lists while recall rewards finding more of a finite relevant set.
- MRR ignores every hit after the first.
- NDCG depends on the gain and discount definitions; variants with the same name can differ.
- Empty-relevance users must not disappear silently from the denominator.
- Offline relevance measures logged behavior under historical exposure, not satisfaction or causal lift.
Practice
Recompute all four metrics for [i4, i3, i5]. Then remove i5 from the candidate set and explain why comparing the two scores would confound ranking quality with candidate construction.
Next steps
Accuracy is only one dimension. Next, measure diversity, coverage, novelty, and serendipity.
Primary sources
Checked 2026-10-17: Järvelin and Kekäläinen, “Cumulated Gain-Based Evaluation of IR Techniques”; Herlocker et al., “Evaluating Collaborative Filtering Recommender Systems”; and Schnabel et al., “Recommendations as Treatments”.