Project: Offline Recommender and Search Evaluator
This capstone evaluates one repository-owned catalog through three visible stages: BM25 retrieval, a small pointwise ranker, and policy-aware re-ranking. It produces evidence, not a hosted search service and not a claim of production impact.
Prerequisites
Complete the preceding 23 lessons, especially online evaluation and feedback loops, temporal splits, ranking metrics, and candidate generation.
Learning goals
- Run one public command without downloads, credentials, or hosted services.
- Inspect tokenization, document frequency, parameters, filters, learned weights, and tie-breaking.
- Compare relevance, coverage, diversity, and constraint violations by stage.
- Reproduce byte-identical evidence and state what it cannot establish.
Fixture and stage contract
Five checked-in files define the catalog, train/evaluation queries, graded judgments, interaction history, and policy. Two training queries fit four pointwise weights. Two disjoint evaluation queries remain untouched until reporting. The interaction feature uses training rows only, but its CTR remains position-biased by construction.
The baseline retrieves and sorts every filter-eligible catalog item with BM25. The ranker combines normalized lexical score, title overlap, freshness, and logged item CTR. The re-ranker enforces safety, availability, one item per provider, and a fresh-result rule at cutoff three. Every tie uses score desc, item_id asc.
Run the public command with an explicit destination:
npm run project:recommender-search-evaluation -- --output /tmp/recommender-search-evidence.json
Worked example
This example runs the command twice in independent temporary directories and compares the bytes before inspecting the served metrics.
import json
import subprocess
import sys
import tempfile
from pathlib import Path
outputs = []
for _ in range(2):
directory = tempfile.TemporaryDirectory()
output = Path(directory.name) / "evidence.json"
subprocess.run(
[sys.executable, "scripts/run-recommender-search-evaluation.py", "--output", str(output)],
check=True,
capture_output=True,
text=True,
)
outputs.append((directory, output.read_bytes()))
evidence = json.loads(outputs[0][1])
metrics = evidence["stages"]["reranked"]["metrics"]
print(f"byte_stable={outputs[0][1] == outputs[1][1]} evaluation_queries={evidence['evaluation_queries']}")
print(f"reranked_metrics={metrics}")
for directory, _ in outputs:
directory.cleanup()
Expected output:
byte_stable=True evaluation_queries=2
reranked_metrics={'catalog_coverage_at_3': 0.8333, 'constraint_violations': 0, 'mean_category_diversity_at_3': 1.0, 'mean_ndcg_at_3': 0.8367}
The evidence shows that baseline BM25 has higher NDCG than the learned stage on this fixture, while the re-ranker removes all encoded violations and raises category diversity. That is a useful failure: the capstone does not cherry-pick a learned “win.” Inspect the rankings and weights, then ablate logged CTR and revise only using training queries.
Re-run the batch contract with:
npm run test:recommender-search-evaluation
It executes the capstone twice, compares exact bytes, runs all eight lesson examples, checks all 24 positions and adjacent links, and audits source and ledger evidence.
Limitations
- Two synthetic evaluation queries cannot estimate production uncertainty, tail behavior, latency, or subgroup quality.
- Complete judgments exist only because the catalog has eight items; real pools contain unjudged documents.
- The simple learner is pedagogical, not a recommended production model.
- Constraint satisfaction validates encoded rules, not the completeness, legality, or fairness of policy.
- Offline gain is not causal online impact; deployment still requires privacy review, safety review, and a powered online test.
Practice
Create an experiment note with one model ablation, one additional query cohort, one policy stress case, and one online-test proposal. Predict which metric should change before running anything, then preserve both confirming and disconfirming evidence.
Next steps
The 24-lesson Recommender Systems, Search & Ranking path is complete. Continue with the site's Graph & Network Data Science path when it is published; do not treat lexical relevance, collaborative preference, and graph proximity as interchangeable signals.
Primary sources
Checked 2026-10-17: NIST TREC, TREC 33 overview; Apache Lucene, BM25Similarity API; Google for Developers, re-ranking; NIST, Privacy Framework.