Skip to main content

Temporal and User-Aware Data Splits

An offline split is a simulation of a future serving decision. If training contains events, aggregates, or features from after that decision, the evaluation leaks information. Random row splits are especially risky when users repeat, items evolve, and actions occur in sequences.

Prerequisites​

You should understand how exposure and selection shape logged feedback.

Learning goals​

  • Use a global time cutoff that matches the intended serving scenario.
  • Keep repeated events and features on the correct side of the cutoff.
  • Report warm users/items separately from cold users/items.
  • Distinguish catalog availability from first observed interaction.

Define the simulation first​

The fixture uses a global time cutoff at 2026-10-01T00:00:00Z. Training contains earlier events; test contains later events. A production-like validation window should sit between them and be used for tuning before the test is opened. In this tiny teaching fixture the explicit split field records only train and test membership so the edge cases stay hand-checkable.

Features must obey the same clock. Popularity at the cutoff uses training events only. A user profile may use history known at request time, not later test actions. An item can be eligible if its item_available_at precedes the request even when its first interaction occurs later.

Worked example: audit warm and cold cohorts​

import csv
from pathlib import Path

path = Path("tests/fixtures/recommender-systems/interactions.csv")
with path.open(encoding="utf-8", newline="") as handle:
rows = list(csv.DictReader(handle))

train = [row for row in rows if row["split"] == "train"]
test = [row for row in rows if row["split"] == "test"]
train_users = {row["user_id"] for row in train}
train_items = {row["item_id"] for row in train}
cold_users = sorted({row["user_id"] for row in test} - train_users)
cold_items = sorted({row["item_id"] for row in test} - train_items)
assert max(row["timestamp"] for row in train) < min(row["timestamp"] for row in test)
print(f"train_rows={len(train)} test_rows={len(test)}")
print(f"cold_users={cold_users} cold_items={cold_items}")

Output:

train_rows=16 test_rows=8
cold_users=['u5'] cold_items=['i7']

A cold user has no training history; a cold item has no training interactions. Those cohorts test different fallbacks. Item i7 was available before the cutoff, so it can legitimately be a candidate even though it has no training interactions.

User-aware alternatives​

A per-user leave-last-one-out split asks a next-interaction question for warm users. It can be useful when that is the declared product task, but users' training and test events may overlap globally in time. It therefore does not simulate a single production cutoff and may use catalog or population evidence from the future. A user holdout tests generalization to entirely unseen users but changes the target population.

Choose the split that matches deployment, then report cohort membership and exclusions. Do not quietly remove users with short histories or items with no positives; that makes the evaluated task easier.

Limitations and common mistakes​

  • Randomly splitting rows from the same user-item sequence leaks later state and duplicates across sets.
  • Computing popularity, encoders, or candidate filters on all rows leaks test information even when labels are split correctly.
  • Removing cold-start cases hides a production failure mode.
  • First interaction time is not necessarily item availability time.
  • One cutoff gives one historical scenario; conclusions may not transfer across seasons or policy changes.

Practice​

Move the cutoff one day later on paper. List which rows, popularity counts, warm/cold labels, and user histories change. Then describe when leave-last-one-out would answer a materially different question.

Next steps​

With the split frozen, compare popularity, recency, and random baselines under identical rules.

Primary sources​

Checked 2026-10-17: Meng et al., “Exploring Data Splitting Strategies for the Evaluation of Recommendation Models”; Sun et al., “A Critical Study on Data Leakage in Recommender System Offline Evaluation”; and Frolov, “Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders”.