Skip to main content

Exposure, Position, and Selection Bias

Observed behavior is downstream of earlier choices. A user can usually act only on an item that was available, retrieved, displayed, and noticed. Honest analysis therefore separates preference from exposure instead of reading the log as a random sample of all user-item pairs.

Prerequisites​

You should understand explicit and implicit feedback and why missing behavior is ambiguous.

Learning goals​

  • Draw the path from eligibility through exposure and examination to action.
  • Distinguish exposure bias, position bias, and selection bias.
  • Explain why naive replay favors the logging policy.
  • State what randomized exposure or propensity methods can and cannot repair.

A measurement path, not a label table​

For a click to appear, several conditions usually hold:

eligible → retrieved → displayed at a position → examined → clicked

Exposure bias arises because the serving policy shows some items more often. Position bias arises because higher positions tend to receive more examination independent of relevance. Selection bias also includes user self-selection, such as rating only memorable items. These mechanisms can coexist.

In the fixture, u1 interacted with i1 at position 1 while i2 received only an impression at position 2. Counting the click as proof that i1 is preferred ignores the different opportunity to be examined. An organic action with exposed=0 came through another route and should not be attributed to the recommendation surface.

Worked example: audit a logged comparison​

Suppose rank 1 had 80 clicks from 1,000 impressions and rank 2 had 40 clicks from 1,000 impressions. The observed click rates are 8% and 4%, but that difference combines item relevance, user mix, presentation, and examination probability. It is not an item-only preference estimate.

If a controlled randomization assigned comparable candidates to positions, examination propensities could support a position-adjusted analysis under stated assumptions. In observational logs, inverse-propensity methods require the exposure probability to be known or defensibly estimated, positive for the actions being evaluated, and correctly connected to the outcome. Very small propensities create high-variance weights.

Feedback loops​

A popular item is retrieved more, gains more impressions and actions, and then appears even more popular. The log increasingly reflects the policy that generated it. Offline evaluation against only observed interactions can reward a new system for reproducing that policy, including its blind spots.

Exploration can create support for learning, but random exposure has user and product costs. It needs eligibility filters, safety constraints, an explicit exploration budget, and monitoring. Debiasing is not a license to expose unsuitable items.

Limitations and common mistakes​

  • Adding rank as a feature does not by itself identify relevance independent of position.
  • Dividing by a propensity is not enough when propensities are wrong, zero, or attached to the wrong decision unit.
  • Treating organic traffic as recommendation exposure misattributes outcomes.
  • Removing popular items does not automatically remove exposure bias.
  • Logged outcomes do not reveal what would have happened under every unserved ranking.

Practice​

Draw a causal diagram with user intent, item relevance, logging policy, position, examination, and click. Identify which arrows make a naive click comparison biased. State one safe randomized intervention and one guardrail.

Next steps​

Next, design temporal and user-aware data splits that do not train on the future or hide cold-start cases.

Primary sources​

Checked 2026-10-17: Schnabel et al., “Recommendations as Treatments: Debiasing Learning and Evaluation”; Joachims, Swaminathan, and Schnabel, “Unbiased Learning-to-Rank with Biased Feedback”; and Google, scoring and positional bias.