Skip to main content

Scoring and Multi-Stage Ranking

A multi-stage recommender spends computation where it matters: fast sources retrieve a manageable pool, a common scorer estimates the declared outcome, and a final re-ranker applies eligibility, diversity, freshness, safety, or exploration constraints.

Prerequisites​

Understand candidate generation, top-k metrics, exposure bias, and beyond-accuracy trade-offs.

Learning goals​

  • Distinguish retrieval scores from a common downstream scoring objective.
  • Keep training labels, feature timestamps, and position effects explicit.
  • Apply a deterministic re-ranking policy after scoring.
  • Budget and evaluate retrieval, scoring, and re-ranking separately.

Model contract​

PartThis lesson's contract
ObjectiveRank retrieved eligible lessons by a declared completion proxy, then apply a bounded cold-item exploration rule.
Input representationCandidate provenance plus user, item, context, and freshness features available at request time.
Learned stateOne calibrated scorer, feature transformations, label window, training policy metadata, and a versioned re-ranking policy.
Inference pathRetrieve, join point-in-time-correct features, score every candidate on one scale, sort, re-rank, and log stage provenance.
Evaluation boundaryMeasure retrieval recall first, then ranking quality conditional on the pool, end-to-end quality, slices, and stage latency.
Principal failure modesTraining-serving skew, leakage, position-biased labels, incomparable source scores, constraint conflicts, and tail latency.

Worked example: score, then constrain​

The toy scorer uses fixed, already-normalized inputs on one scale: 0.4 * content + 0.4 * collaborative + 0.2 * completion. These values are illustrative learned-state inputs, not performance measurements. The policy then uses one declared cold-item exploration slot while requiring a validated warm item first.

features = {
"i1": (0.1, 0.1, 0.1),
"i4": (0.8, 0.7, 0.7),
"i5": (0.3, 0.2, 0.3),
"i7": (0.8, 0.8, 0.6),
}
weights = (0.4, 0.4, 0.2)
scored = sorted(
((item, sum(weight * value for weight, value in zip(weights, values))) for item, values in features.items()),
key=lambda pair: (-pair[1], pair[0]),
)
cold_items = {"i7"}
warm = [item for item, _ in scored if item not in cold_items]
cold = [item for item, _ in scored if item in cold_items]
served = warm[:1] + cold[:1] + warm[1:2]
budget_ms = {"retrieve": 20, "score": 35, "rerank": 5}
print("scored=[" + ", ".join(f"('{item}', {score:.3f})" for item, score in scored) + "]")
print(f"served={served} budget_ms={budget_ms}")

Output:

scored=[('i7', 0.760), ('i4', 0.740), ('i5', 0.260), ('i1', 0.100)]
served=['i4', 'i7', 'i5'] budget_ms={'retrieve': 20, 'score': 35, 'rerank': 5}

The 60 ms latency budget is an allocation, not a measured claim. Production evidence needs percentile, load, hardware, timeout, and fallback behavior. The re-ranking rule also expresses a product choice: it may improve item learning while reducing immediate ranking quality, so report both the pre-policy score order and served order.

Evaluation by stage​

Use the shared fixture and constraints for comparisons; do not create a favorable candidate set per model. Report retrieval recall over the full eligible catalog, ranking quality both conditional on the retrieved pool and end to end, beyond-accuracy and stakeholder measures, and latency per stage. Model comparisons are conditional on this protocol—there is no universal model ranking.

The label window and features must be point-in-time correct. Historical position affects which labels are observed. Train-only debiasing assumptions, propensity support, and any sampled negatives belong in the model record. An offline metric is not product impact; online experiments or another credible causal design remain necessary.

Limitations and common mistakes​

  • Candidate-generator scores from different sources are not automatically comparable.
  • A complex scorer cannot recover relevant items absent from retrieval.
  • Re-ranking can violate score calibration and must be evaluated on the served order.
  • Hard rules can conflict or empty the list; define priority and fallback explicitly.
  • Average latency hides tail failures, while average ranking quality hides cold and exposure-defined cohorts.

Practice​

Swap the cold item into the first position and name the intended learning benefit, user risk, guardrail, and online randomization unit. Then reduce scoring to 20 ms and state what capacity or feature trade-off you would measure.

Next steps​

Search relevance and judgment data connects recommendation ranking to lexical retrieval and explicit relevance evidence.

Primary sources​

Checked 2026-10-17: Google, recommendation-system stages, scoring guidance, and re-ranking guidance; Covington, Adams, and Sargin, “Deep Neural Networks for YouTube Recommendations”.