Skip to main content

Recommendation Problems and Product Objectives

A recommender does not merely predict what a person might click. It chooses a small, ordered set of items from an eligible candidate set, for a particular user and context, in order to support a product decision. The objective determines what evidence is useful and what failure means.

Prerequisites​

You should be comfortable with rows, columns, proportions, and the difference between prediction and causation. The supervised-learning introduction is useful background, but no recommender-system experience is required.

Learning goals​

  • Name the user or decision unit, item, context, eligible candidate set, action, and outcome.
  • Translate a product goal into an offline prediction target without claiming that the target is product impact.
  • Pair a primary objective with guardrails and a measurement horizon.
  • Recognize when recommendation, retrieval, and re-ranking are separate stages.

Start with the decision​

Imagine a learning app that can show three lessons on a home screen. “Recommend useful lessons” is not yet an operational objective. A decision record needs at least:

QuestionConcrete answer
Decision unitone eligible learner visit
Candidatespublished lessons the learner may access and has not completed
Actionchoose and order three lessons
Proximal outcomea lesson start within the visit
Longer-horizon outcomea completed lesson within seven days
Guardrailshide rate, topic coverage, latency, and accessibility defects
Retraining/evaluation timetrain through 2026-09-30; evaluate later visits

This contract prevents a metric from silently changing meaning. Click-through rate, completion rate, and retained learning answer different questions. Optimizing only clicks can reward tempting titles without helping learners finish; optimizing only total watch or dwell time can reward long items.

Worked example: write an objective card​

For the lesson surface, use this objective card before choosing a model:

For eligible returning learners, rank three available and unfinished lessons at visit time to increase seven-day lesson completion, while monitoring immediate starts, hides, subject coverage, page latency, and accessibility errors.

The card identifies the action and observation window. It does not say that a high offline ranking score causes more completions. Historical outcomes were generated by earlier interfaces and policies; a product-impact claim requires an online experiment or another defensible causal design.

From objective to evaluation protocol​

An offline protocol is a deliberately limited proxy. It must freeze:

  1. the training cutoff and test horizon;
  2. the eligible candidate set available at each decision time;
  3. the observed event that counts as relevance;
  4. the list cutoff, such as k = 3;
  5. user inclusion and aggregation rules; and
  6. baselines and guardrails reported beside the primary metric.

Later pages keep those choices visible. Candidate generation, scoring, and re-ranking can optimize different proxies, so scores from different stages are not automatically comparable.

Limitations and common mistakes​

  • Proxy becomes the goal. A click is observable, but it is not automatically satisfaction, learning, or incremental value.
  • Candidates are left implicit. Ranking only items that happened to receive feedback creates a different, easier task than ranking all eligible items.
  • Stakeholders disappear. Users, item providers, and the platform can have conflicting outcomes; one average cannot represent all of them.
  • Offline lift becomes causal language. Historical replay compares behavior under logged exposure. It does not establish the outcome under a new serving policy.
  • No guardrail or horizon is named. The system can improve a short-term proxy while harming coverage, latency, or longer-term outcomes.

Practice​

Choose one surface you know. Write its decision unit, candidate-set rule, list size, immediate outcome, longer-horizon outcome, two guardrails, training cutoff, and test horizon. Then underline every word that would require an event or catalog field.

Next steps​

Next, model those fields explicitly as users, items, context, and feedback. The planned online-evaluation-and-feedback-loops lesson will later revisit why offline relevance is not product impact.

Primary sources​

Checked 2026-10-17: Google's recommendation-system architecture overview, Google's guidance on scoring objectives and positional bias, and Schnabel et al., “Recommendations as Treatments: Debiasing Learning and Evaluation”.