Skip to main content

BM25 and Field-Aware Scoring

BM25 rewards documents that contain query terms, discounts terms that occur in many documents, saturates repeated term frequency, and normalizes for document length. It is a strong lexical baseline because every component is inspectable.

Prerequisites​

Understand inverted indexes, logarithms, term frequency, and document frequency.

Learning goals​

  • Read every part of the BM25 scoring expression.
  • Explain the roles of k1k_1 and bb.
  • Keep title and description evidence observable.
  • Tune retrieval parameters only on training or validation queries.

Scoring contract​

For query term tt and document dd, this batch uses Lucene's positive IDF form and the BM25 term contribution

idf⁡(t)=log⁡(1+N−df(t)+0.5df(t)+0.5),\operatorname{idf}(t)=\log\left(1+\frac{N-df(t)+0.5}{df(t)+0.5}\right), s(t,d)=idf⁡(t)tf(t,d)(k1+1)tf(t,d)+k1(1−b+b∣d∣∣d∣‾).s(t,d)=\operatorname{idf}(t)\frac{tf(t,d)(k_1+1)}{tf(t,d)+k_1\left(1-b+b\frac{|d|}{\overline{|d|}}\right)}.

The executable contract fixes k1 = 1.2, b = 0.75, a title boost of 2.0, and the tie-break score desc, item_id asc. Larger k1k_1 delays term-frequency saturation. Setting b=0b=0 disables length normalization; b=1b=1 applies its full BM25 form. These are parameters, not facts about every corpus.

Worked example​

The example calls the same repository implementation used by the capstone and exposes the scores before any learned ranker or policy re-ranking.

import runpy

module = runpy.run_path("scripts/run-recommender-search-evaluation.py")
catalog = module["read_csv"]("search-catalog.csv")
index, df, average_length = module["build_index"](catalog)
tokens = module["tokenize"]("lexical search ranking")
scores = []
for item in catalog:
score = module["bm25_score"](tokens, index[item["item_id"]], df, len(catalog), average_length)
scores.append((item["item_id"], round(score, 4)))
print(f"k1=1.2 b=0.75 title_boost=2.0 avgdl={average_length:.1f}")
print(f"top3={sorted(scores, key=lambda pair: (-pair[1], pair[0]))[:3]}")

Expected output:

k1=1.2 b=0.75 title_boost=2.0 avgdl=10.0
top3=[('s4', 3.0622), ('s3', 2.1309), ('s6', 2.0337)]

The title boost is an explicit product hypothesis: a title match is treated like repeated evidence. In larger systems, BM25F-style combined-field scoring can normalize fields more carefully. Whatever form you use, log the analyzer version, field mapping, boosts, parameters, and index snapshot with evaluation results.

Limitations​

  • BM25 measures lexical evidence, not truth, safety, authority, or user satisfaction.
  • Field weights can overfit a small query set and can amplify title spam.
  • Length is only a rough nuisance variable; short is not automatically better.
  • Scores are not calibrated probabilities and are not comparable across arbitrary indexes or query formulations.

Practice​

Set b=0b=0, recompute the ranking, and explain which documents move. Then propose a validation-query protocol for choosing a title boost without looking at evaluation-query judgments.

Next steps​

Query understanding, filters, and facets separates scored text evidence from exact eligibility and exploration controls.

Primary sources​

Checked 2026-10-17: Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond; Apache Lucene, BM25Similarity API; Apache Lucene, field-aware scoring guidance.