Recommendation Problems and Product Objectives
A recommender does not merely predict what a person might click. It chooses a small, ordered set of items from an eligible candidate set, for a particular user and context, in order to support a product decision. The objective determines what evidence is useful and what failure means.
Prerequisites
You should be comfortable with rows, columns, proportions, and the difference between prediction and causation. The supervised-learning introduction is useful background, but no recommender-system experience is required.
Learning goals
- Name the user or decision unit, item, context, eligible candidate set, action, and outcome.
- Translate a product goal into an offline prediction target without claiming that the target is product impact.
- Pair a primary objective with guardrails and a measurement horizon.
- Recognize when recommendation, retrieval, and re-ranking are separate stages.
Start with the decision
Imagine a learning app that can show three lessons on a home screen. “Recommend useful lessons” is not yet an operational objective. A decision record needs at least:
| Question | Concrete answer |
|---|---|
| Decision unit | one eligible learner visit |
| Candidates | published lessons the learner may access and has not completed |
| Action | choose and order three lessons |
| Proximal outcome | a lesson start within the visit |
| Longer-horizon outcome | a completed lesson within seven days |
| Guardrails | hide rate, topic coverage, latency, and accessibility defects |
| Retraining/evaluation time | train through 2026-09-30; evaluate later visits |
This contract prevents a metric from silently changing meaning. Click-through rate, completion rate, and retained learning answer different questions. Optimizing only clicks can reward tempting titles without helping learners finish; optimizing only total watch or dwell time can reward long items.
Worked example: write an objective card
For the lesson surface, use this objective card before choosing a model:
For eligible returning learners, rank three available and unfinished lessons at visit time to increase seven-day lesson completion, while monitoring immediate starts, hides, subject coverage, page latency, and accessibility errors.
The card identifies the action and observation window. It does not say that a high offline ranking score causes more completions. Historical outcomes were generated by earlier interfaces and policies; a product-impact claim requires an online experiment or another defensible causal design.
From objective to evaluation protocol
An offline protocol is a deliberately limited proxy. It must freeze:
- the training cutoff and test horizon;
- the eligible candidate set available at each decision time;
- the observed event that counts as relevance;
- the list cutoff, such as
k = 3; - user inclusion and aggregation rules; and
- baselines and guardrails reported beside the primary metric.
Later pages keep those choices visible. Candidate generation, scoring, and re-ranking can optimize different proxies, so scores from different stages are not automatically comparable.
Limitations and common mistakes
- Proxy becomes the goal. A click is observable, but it is not automatically satisfaction, learning, or incremental value.
- Candidates are left implicit. Ranking only items that happened to receive feedback creates a different, easier task than ranking all eligible items.
- Stakeholders disappear. Users, item providers, and the platform can have conflicting outcomes; one average cannot represent all of them.
- Offline lift becomes causal language. Historical replay compares behavior under logged exposure. It does not establish the outcome under a new serving policy.
- No guardrail or horizon is named. The system can improve a short-term proxy while harming coverage, latency, or longer-term outcomes.
Practice
Choose one surface you know. Write its decision unit, candidate-set rule, list size, immediate outcome, longer-horizon outcome, two guardrails, training cutoff, and test horizon. Then underline every word that would require an event or catalog field.
Next steps
Next, model those fields explicitly as users, items, context, and feedback. The planned online-evaluation-and-feedback-loops lesson will later revisit why offline relevance is not product impact.
Primary sources
Checked 2026-10-17: Google's recommendation-system architecture overview, Google's guidance on scoring objectives and positional bias, and Schnabel et al., “Recommendations as Treatments: Debiasing Learning and Evaluation”.