Re-ranking with Business Constraints
A relevance score is not a complete serving decision. The final stage must enforce eligibility and may trade some predicted relevance for diversity, freshness, risk control, inventory health, or other declared product goals.
Prerequisites
Understand learning to rank, candidate provenance, and beyond-accuracy metrics.
Learning goals
- Separate hard constraints from soft preferences.
- Make safety, availability, freshness, diversity, and policy rules explicit.
- Record pre-policy and served orders plus rejection reasons.
- Measure the relevance and coverage cost of re-ranking.
Constraint contract
The local policy requires safe and available items, at most one result per provider, and at least one result published on or after 2026-10-01 in the top three. Safety and availability are hard filters. Provider diversity and freshness are explicit list constraints. A production owner must define precedence and fallback when the candidate pool cannot satisfy every rule.
Do not hide policy inside a single blended score. Hard rules need auditable pass/fail reasons. Soft preferences need weights, units, and a recorded trade-off curve. Keep the original score order so relevance loss, constraint repair, and displaced items remain observable.
Worked example
The learned order begins with an unsafe item and contains two items from the same provider. A deterministic greedy pass applies the checked-in policy.
import json
import runpy
module = runpy.run_path("scripts/run-recommender-search-evaluation.py")
catalog = module["read_csv"]("search-catalog.csv")
items = {item["item_id"]: item for item in catalog}
with open("tests/fixtures/recommender-systems/search-policy.json", encoding="utf-8") as handle:
policy = json.load(handle)
ranked = ["s5", "s4", "s3", "s7", "s1", "s8", "s2", "s6"]
served = module["rerank"](ranked, items, policy)
print(f"pre_policy_top3={ranked[:3]} violations={module['violations'](ranked[:3], items, policy)}")
print(f"served={served} violations={module['violations'](served, items, policy)}")
Expected output:
pre_policy_top3=['s5', 's4', 's3'] violations=2
served=['s4', 's7', 's1'] violations=0
Report NDCG, catalog coverage, category diversity, and violations for baseline, ranked, and served stages. A zero violation count does not prove that the policy is fair, complete, or harmless. It only proves that this implementation satisfied the encoded rules on this fixture.
Limitations
- Greedy selection can miss a feasible or higher-utility list that a constrained optimizer would find.
- Metadata can be stale, missing, or strategically manipulated.
- Provider diversity is not a substitute for user-group fairness or viewpoint quality.
- Freshness can crowd out durable content; safety classifiers can produce harmful false positives and negatives.
- Policy changes need versioning, review, rollback, and monitoring.
Practice
Construct a candidate pool in which the top-three cutoff and provider cap are infeasible together. Define whether to return fewer results, relax a soft rule, or invoke a safe fallback, and record who owns that decision.
Next steps
Online evaluation and feedback loops explains why served behavior still requires bias-aware logging and controlled experiments.
Primary sources
Checked 2026-10-17: Google for Developers, re-ranking and recommendation-system stages; Carbonell and Goldstein, The Use of MMR, Diversity-Based Reranking for Reordering Documents.