Popularity, Recency, and Random Baselines
A complex model is informative only when it beats relevant simple policies under the same task. Popularity, recency, and seeded random rankings expose whether personalization adds value beyond frequent, fresh, or chance ordering.
Prerequisites
Freeze the temporal and user-aware split, relevance event, evaluation users, and catalog clock first.
Learning goals
- Define three reproducible baselines without using test outcomes.
- Compare every method under an identical candidate set and cutoff.
- Use deterministic tie-breaking and a fixed random seed.
- Interpret what each baseline controls for and what it does not.
Baseline definitions
At the training cutoff:
- Popularity ranks by the number of positive training actions, then by item ID for ties.
- Recency ranks by the latest positive training action, then by item ID.
- Random shuffles the same ordered candidates using a recorded seed.
For u1, the catalog contains items available by the evaluation time. We remove items already seen by u1 in training. All three methods receive the resulting identical candidate set. The example counts action events deliberately; another protocol might count unique users or requests, but it must say so.
Worked example: compare the rankings
import csv
import random
from collections import Counter
from pathlib import Path
path = Path("tests/fixtures/recommender-systems/interactions.csv")
with path.open(encoding="utf-8", newline="") as handle:
rows = list(csv.DictReader(handle))
train = [row for row in rows if row["split"] == "train"]
positive = {"click", "save", "complete", "rating"}
seen = {row["item_id"] for row in train if row["user_id"] == "u1"}
catalog = {row["item_id"] for row in rows if row["item_available_at"] <= "2026-10-01T09:00:00Z"}
candidates = sorted(catalog - seen)
counts = Counter(row["item_id"] for row in train if row["event_type"] in positive)
latest = {
item: max(row["timestamp"] for row in train if row["item_id"] == item and row["event_type"] in positive)
for item in candidates if counts[item]
}
popularity = sorted(candidates, key=lambda item: (-counts[item], item))[:3]
recency = sorted(candidates, key=lambda item: (latest.get(item, ""), item), reverse=True)[:3]
random_rank = candidates.copy()
random.Random(7).shuffle(random_rank)
print(f"candidates={candidates}")
print(f"popularity={popularity}")
print(f"recency={recency}")
print(f"random(seed=7)={random_rank[:3]}")
Output:
candidates=['i3', 'i4', 'i5', 'i6', 'i7']
popularity=['i4', 'i3', 'i5']
recency=['i5', 'i4', 'i3']
random(seed=7)=['i7', 'i3', 'i6']
What a baseline reveals
If a personalized method cannot beat popularity for the declared metric, its extra complexity is not justified by that experiment. A recency win may show a rapidly changing catalog or temporal target. Random provides a sanity floor and catches candidate or metric bugs, but it is not necessarily safe to serve without eligibility constraints.
These rankings are not interchangeable interventions. Popularity can concentrate exposure; recency can starve durable items; random can reduce immediate relevance. Report coverage and cohort behavior, not only one accuracy number.
Limitations and common mistakes
- Using test interactions to compute popularity or recency leaks the answer.
- Giving different methods different candidates confounds ranking with retrieval.
- Unspecified tie-breaking makes results irreproducible.
- Counting repeated actions can let a few users dominate; unique-user popularity answers a different question.
- A baseline win offline is not proof of incremental product impact.
Practice
Replace action-count popularity with unique-user popularity. Predict which ties change, then calculate the ranking. State which definition better matches a home-feed objective and why.
Next steps
Next, score these lists with explicitly defined top-k ranking metrics.
Primary sources
Checked 2026-10-17: Google, candidate generation and its biases; Google, recommendation scoring; and Herlocker et al., “Evaluating Collaborative Filtering Recommender Systems”.