BM25 and Field-Aware Scoring
BM25 rewards documents that contain query terms, discounts terms that occur in many documents, saturates repeated term frequency, and normalizes for document length. It is a strong lexical baseline because every component is inspectable.
Prerequisites
Understand inverted indexes, logarithms, term frequency, and document frequency.
Learning goals
- Read every part of the BM25 scoring expression.
- Explain the roles of and .
- Keep title and description evidence observable.
- Tune retrieval parameters only on training or validation queries.
Scoring contract
For query term and document , this batch uses Lucene's positive IDF form and the BM25 term contribution
The executable contract fixes k1 = 1.2, b = 0.75, a title boost of 2.0, and the tie-break score desc, item_id asc. Larger delays term-frequency saturation. Setting disables length normalization; applies its full BM25 form. These are parameters, not facts about every corpus.
Worked example
The example calls the same repository implementation used by the capstone and exposes the scores before any learned ranker or policy re-ranking.
import runpy
module = runpy.run_path("scripts/run-recommender-search-evaluation.py")
catalog = module["read_csv"]("search-catalog.csv")
index, df, average_length = module["build_index"](catalog)
tokens = module["tokenize"]("lexical search ranking")
scores = []
for item in catalog:
score = module["bm25_score"](tokens, index[item["item_id"]], df, len(catalog), average_length)
scores.append((item["item_id"], round(score, 4)))
print(f"k1=1.2 b=0.75 title_boost=2.0 avgdl={average_length:.1f}")
print(f"top3={sorted(scores, key=lambda pair: (-pair[1], pair[0]))[:3]}")
Expected output:
k1=1.2 b=0.75 title_boost=2.0 avgdl=10.0
top3=[('s4', 3.0622), ('s3', 2.1309), ('s6', 2.0337)]
The title boost is an explicit product hypothesis: a title match is treated like repeated evidence. In larger systems, BM25F-style combined-field scoring can normalize fields more carefully. Whatever form you use, log the analyzer version, field mapping, boosts, parameters, and index snapshot with evaluation results.
Limitations
- BM25 measures lexical evidence, not truth, safety, authority, or user satisfaction.
- Field weights can overfit a small query set and can amplify title spam.
- Length is only a rough nuisance variable; short is not automatically better.
- Scores are not calibrated probabilities and are not comparable across arbitrary indexes or query formulations.
Practice
Set , recompute the ranking, and explain which documents move. Then propose a validation-query protocol for choosing a title boost without looking at evaluation-query judgments.
Next steps
Query understanding, filters, and facets separates scored text evidence from exact eligibility and exploration controls.
Primary sources
Checked 2026-10-17: Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond; Apache Lucene, BM25Similarity API; Apache Lucene, field-aware scoring guidance.