Learning to Rank
Learning to rank (LTR) estimates an ordering function from query-item features and relevance evidence. Retrieval still determines which items can be ranked; a ranker cannot recover a relevant item that never entered its candidate set.
Prerequisites
Understand query and filter contracts, graded judgments, temporal query splits, and NDCG.
Learning goals
- Distinguish pointwise, pairwise, and listwise training instances and losses.
- Keep the training objective separate from the final ranked-list metric.
- Build features using only information available at request time.
- Diagnose biased feedback and query-level leakage.
Three objective families
Pointwise methods treat each query-item row as a regression or classification example. They are simple, but the loss does not directly represent ordering within a query. Pairwise methods learn which of two items should rank higher for the same query; RankNet is a classic example. Listwise methods treat an entire query's candidate list as the training instance and define a loss over its ordering or score distribution.
These are objective families, not evaluation metrics. A pointwise model, a pairwise model, and a listwise model may all be judged by NDCG@k, recall@k, latency, and policy metrics on the same frozen evaluation queries. Do not claim that optimizing one surrogate guarantees improvement in the product metric.
Worked example
The fixture exposes the different training units without pretending to train three comparable models.
import csv
with open("tests/fixtures/recommender-systems/search-judgments.csv", encoding="utf-8", newline="") as handle:
grades = {row["item_id"]: int(row["grade"]) for row in csv.DictReader(handle) if row["query_id"] == "q2"}
pointwise_rows = sorted(grades.items())
pairwise_preferences = [
(left, right) for left, left_grade in pointwise_rows for right, right_grade in pointwise_rows
if left_grade > right_grade
]
listwise_target = sorted(pointwise_rows, key=lambda pair: (-pair[1], pair[0]))
print(f"pointwise_rows={len(pointwise_rows)} pairwise_preferences={len(pairwise_preferences)}")
print(f"listwise_target={listwise_target}")
Expected output:
pointwise_rows=8 pairwise_preferences=21
listwise_target=[('s3', 3), ('s7', 2), ('s2', 1), ('s4', 1), ('s1', 0), ('s5', 0), ('s6', 0), ('s8', 0)]
The capstone fits a deliberately small pointwise linear scorer on the two training queries. Its features are normalized BM25, title overlap, freshness, and logged item CTR. The weights are evidence about that fixture only. Because CTR inherits exposure and position bias, it can make the learned ranking worse than BM25 on evaluation queries; that failure is reported, not hidden.
Split by query and time before feature fitting and hyperparameter choice. Compute aggregates, encoders, and click features from training history only. Evaluate both retrieval recall and ranker quality, because conditioning evaluation on retrieved candidates otherwise hides missed relevant items.
Limitations
- Judgment labels may be sparse, subjective, policy-dependent, or pooled from earlier systems.
- Pair construction can overrepresent queries with many judged candidates.
- Listwise losses can be expensive and still optimize a surrogate.
- Logged behavior contains exposure, position, trust, and selection bias; propensity correction requires support and a defensible examination model.
Practice
Remove logged CTR from the capstone feature set and compare evaluation NDCG. Explain why this ablation diagnoses one feature on one fixture rather than establishing a universal ranking rule.
Next steps
Re-ranking with business constraints converts a score order into an explicitly governed served list.
Primary sources
Checked 2026-10-17: Burges et al., Learning to Rank using Gradient Descent; Cao et al., Learning to Rank: From Pairwise Approach to Listwise Approach; Joachims, Swaminathan, and Schnabel, Unbiased Learning-to-Rank with Biased Feedback.