Search Relevance and Judgment Data
Search starts with a question, a corpus, and an action the reader should be able to take. “Relevant” is not an intrinsic document label. It is a judgment about whether a particular result helps with a particular information need under a stated rubric.
Prerequisites
Understand scoring and multi-stage ranking, top-k metrics, temporal splits, and exposure bias.
Learning goals
- Write a query-level relevance rubric before judging results.
- Preserve graded judgments and assessor provenance.
- Distinguish unjudged documents from documents judged non-relevant.
- Separate reusable offline evidence from causal product claims.
A judgment contract
The local fixture uses grades 0–3: 0 is judged non-relevant, 1 is marginal, 2 is useful, and 3 directly satisfies the intent. A production rubric also needs query context, locale, eligibility rules, assessor instructions, adjudication, and a policy for disagreement. Store the rubric version with every judgment set.
TREC-style pools combine top results from multiple runs for assessment. Pooling makes evaluation feasible, but it does not make the remaining corpus irrelevant. Unjudged is not irrelevant. Coverage depends on the systems that contributed to the pool, and shallow pools can systematically miss relevant result types.
Clicks are not drop-in relevance labels. A click requires exposure and examination, is affected by rank and presentation, and may reflect curiosity rather than satisfaction. Editorial judgments have their own subjectivity, but their task and scale can be inspected directly.
Worked example
The repository judgment fixture is complete only for this synthetic eight-item catalog. It keeps query and item identifiers separate from the grade.
import csv
path = "tests/fixtures/recommender-systems/search-judgments.csv"
with open(path, encoding="utf-8", newline="") as handle:
rows = list(csv.DictReader(handle))
q3 = sorted(
((row["item_id"], int(row["grade"])) for row in rows if row["query_id"] == "q3"),
key=lambda pair: (-pair[1], pair[0]),
)
print(f"judgments={len(rows)} q3_graded={q3[:4]}")
print("unjudged_policy=keep-separate-from-grade-0")
Expected output:
judgments=32 q3_graded=[('s4', 3), ('s3', 2), ('s7', 1), ('s1', 0)]
unjudged_policy=keep-separate-from-grade-0
Use a query split before fitting a ranker. Item-level random splitting lets the same query intent influence training and evaluation and overstates generalization to new needs.
Limitations
- A tiny synthetic catalog cannot measure assessor agreement or long-tail query coverage.
- A grade compresses usefulness, correctness, authority, freshness, and safety into one ordinal value unless the rubric keeps those axes separate.
- Reusing judgments after the corpus, intent distribution, or policy changes creates version mismatch.
- Offline gain is not causal online impact; it is evidence about one frozen test collection.
Practice
Write two competing intents for “python ranking.” Define what grade 3 means for each, identify which context disambiguates them, and describe how you would sample results outside the current pool.
Next steps
Inverted indexes and lexical retrieval turns the catalog into an inspectable candidate generator.
Primary sources
Checked 2026-10-17: NIST TREC, TREC 33 overview: relevance judgments and pooling; NIST TREC, English relevance judgments; Joachims, Swaminathan, and Schnabel, Unbiased Learning-to-Rank with Biased Feedback.