Online Evaluation and Feedback Loops
Offline evaluation asks how a frozen system ranks a frozen judged set. Online evaluation asks what happens when a serving policy changes what people see and can do. The second question is causal and needs a controlled design, guardrails, and operational ownership.
Prerequisites
Understand constraint-aware re-ranking, exposure bias, randomized experiments, and privacy risk.
Learning goals
- Log exposure, position, policy, and outcome without equating clicks with preference.
- Recognize feedback loops created by ranking and data collection.
- Design an online test with predeclared metrics and guardrails.
- Apply privacy purpose, access, retention, and deletion boundaries.
From serving to evidence
An impression record should identify the request, candidate or served item, rank, policy/model versions, eligibility decision, timestamp, and experiment assignment. Outcomes need a declared window and join contract. Missing outcomes are not automatically negative preference.
Position bias occurs because higher-ranked items are more likely to be examined. Trust, snippet, device, and latency effects also influence action. Propensity weighting can adjust some estimands only when exposure probabilities are known or defensibly estimated and every evaluated action has support. It is not a generic “debias” switch.
Worked example
The training log deliberately has higher observed CTR at earlier positions. The calculation diagnoses the log; it does not estimate intrinsic relevance.
import csv
from collections import Counter
views = Counter()
clicks = Counter()
with open("tests/fixtures/recommender-systems/search-interactions.csv", encoding="utf-8", newline="") as handle:
for row in csv.DictReader(handle):
if row["split"] == "train":
views[row["position"]] += 1
clicks[row["position"]] += int(row["clicked"])
rates = {position: round(clicks[position] / count, 3) for position, count in sorted(views.items())}
print(f"observed_ctr_by_position={rates}")
print("interpretation=exposure-conditioned-observation-not-preference")
Expected output:
observed_ctr_by_position={'1': 1.0, '2': 0.5, '3': 0.0}
interpretation=exposure-conditioned-observation-not-preference
A ranking policy changes exposure; exposure generates future training data; the next model can reinforce the policy's earlier choices. Monitor catalog and provider exposure, query cohorts, zero-result rates, rejected-item reasons, long-term satisfaction, and exploration support—not clicks alone.
For an online test, state the unit of randomization, eligibility population, primary metric, guardrails, minimum detectable effect, duration, novelty and carryover risks, multiple-testing plan, and abort rule before launch. Preserve assignment and analyze by assigned variant. Offline gain is not causal online impact, and an online test result may still fail to generalize beyond its population and time window.
Privacy begins before logging. Record a legitimate purpose, minimize fields and precision, restrict access, set retention and deletion behavior, protect identifiers, and assess whether query text can contain sensitive information. Consent and legal requirements depend on jurisdiction and use; this lesson is not legal advice.
Limitations
- Short tests miss delayed satisfaction, ecosystem effects, and supplier adaptation.
- Interference can violate independent-unit assumptions when users or items affect one another.
- Novelty, seasonality, outages, and concurrent launches can distort results.
- Privacy and safety harms may not be captured by aggregate engagement guardrails.
Practice
Draft an experiment card for the re-ranking policy: primary metric, two guardrails, randomization unit, support requirement, stop condition, retention period, and named owner.
Next steps
Project: offline recommender and search evaluator assembles the local retrieval, ranking, constraint, and evidence contracts.
Primary sources
Checked 2026-10-17: Joachims, Swaminathan, and Schnabel, Unbiased Learning-to-Rank with Biased Feedback; Kohavi et al., Controlled experiments on the web; NIST, Privacy Framework.