Scoring and Multi-Stage Ranking
A multi-stage recommender spends computation where it matters: fast sources retrieve a manageable pool, a common scorer estimates the declared outcome, and a final re-ranker applies eligibility, diversity, freshness, safety, or exploration constraints.
Prerequisites
Understand candidate generation, top-k metrics, exposure bias, and beyond-accuracy trade-offs.
Learning goals
- Distinguish retrieval scores from a common downstream scoring objective.
- Keep training labels, feature timestamps, and position effects explicit.
- Apply a deterministic re-ranking policy after scoring.
- Budget and evaluate retrieval, scoring, and re-ranking separately.
Model contract
| Part | This lesson's contract |
|---|---|
| Objective | Rank retrieved eligible lessons by a declared completion proxy, then apply a bounded cold-item exploration rule. |
| Input representation | Candidate provenance plus user, item, context, and freshness features available at request time. |
| Learned state | One calibrated scorer, feature transformations, label window, training policy metadata, and a versioned re-ranking policy. |
| Inference path | Retrieve, join point-in-time-correct features, score every candidate on one scale, sort, re-rank, and log stage provenance. |
| Evaluation boundary | Measure retrieval recall first, then ranking quality conditional on the pool, end-to-end quality, slices, and stage latency. |
| Principal failure modes | Training-serving skew, leakage, position-biased labels, incomparable source scores, constraint conflicts, and tail latency. |
Worked example: score, then constrain
The toy scorer uses fixed, already-normalized inputs on one scale: 0.4 * content + 0.4 * collaborative + 0.2 * completion. These values are illustrative learned-state inputs, not performance measurements. The policy then uses one declared cold-item exploration slot while requiring a validated warm item first.
features = {
"i1": (0.1, 0.1, 0.1),
"i4": (0.8, 0.7, 0.7),
"i5": (0.3, 0.2, 0.3),
"i7": (0.8, 0.8, 0.6),
}
weights = (0.4, 0.4, 0.2)
scored = sorted(
((item, sum(weight * value for weight, value in zip(weights, values))) for item, values in features.items()),
key=lambda pair: (-pair[1], pair[0]),
)
cold_items = {"i7"}
warm = [item for item, _ in scored if item not in cold_items]
cold = [item for item, _ in scored if item in cold_items]
served = warm[:1] + cold[:1] + warm[1:2]
budget_ms = {"retrieve": 20, "score": 35, "rerank": 5}
print("scored=[" + ", ".join(f"('{item}', {score:.3f})" for item, score in scored) + "]")
print(f"served={served} budget_ms={budget_ms}")
Output:
scored=[('i7', 0.760), ('i4', 0.740), ('i5', 0.260), ('i1', 0.100)]
served=['i4', 'i7', 'i5'] budget_ms={'retrieve': 20, 'score': 35, 'rerank': 5}
The 60 ms latency budget is an allocation, not a measured claim. Production evidence needs percentile, load, hardware, timeout, and fallback behavior. The re-ranking rule also expresses a product choice: it may improve item learning while reducing immediate ranking quality, so report both the pre-policy score order and served order.
Evaluation by stage
Use the shared fixture and constraints for comparisons; do not create a favorable candidate set per model. Report retrieval recall over the full eligible catalog, ranking quality both conditional on the retrieved pool and end to end, beyond-accuracy and stakeholder measures, and latency per stage. Model comparisons are conditional on this protocol—there is no universal model ranking.
The label window and features must be point-in-time correct. Historical position affects which labels are observed. Train-only debiasing assumptions, propensity support, and any sampled negatives belong in the model record. An offline metric is not product impact; online experiments or another credible causal design remain necessary.
Limitations and common mistakes
- Candidate-generator scores from different sources are not automatically comparable.
- A complex scorer cannot recover relevant items absent from retrieval.
- Re-ranking can violate score calibration and must be evaluated on the served order.
- Hard rules can conflict or empty the list; define priority and fallback explicitly.
- Average latency hides tail failures, while average ranking quality hides cold and exposure-defined cohorts.
Practice
Swap the cold item into the first position and name the intended learning benefit, user risk, guardrail, and online randomization unit. Then reduce scoring to 20 ms and state what capacity or feature trade-off you would measure.
Next steps
Search relevance and judgment data connects recommendation ranking to lexical retrieval and explicit relevance evidence.
Primary sources
Checked 2026-10-17: Google, recommendation-system stages, scoring guidance, and re-ranking guidance; Covington, Adams, and Sargin, “Deep Neural Networks for YouTube Recommendations”.