Skip to main content

Offline–Online Feature Consistency

Training–serving skew occurs when the feature values or transformations used during serving do not mean the same thing as those used to fit and evaluate the model. Matching field names are insufficient.

Prerequisites​

Complete Latency, Throughput, and Cost. Know which Study Sprint preprocessing state was learned from the training partition and serialized in model.json.

Learning goals​

  • distinguish transformation, source, timing, and data-distribution differences.
  • preserve learned preprocessing as model state rather than recomputing it on requests.
  • test parity at the public batch and online seams with identical raw instances.

Four consistency questions​

  1. Definition: Does practice_minutes count the same activity, unit, and population everywhere?
  2. Event time: Could the value have been known at the prediction timestamp, or does a historical join leak the future?
  3. Transformation: Are missingness, scaling, category mapping, clipping, and ordering identical?
  4. Freshness: Which observation time and materialization time does an online value represent?

A feature store can help manage historical retrieval, materialization, and definitions, but adopting one does not automatically correct bad timestamps, labels, or ownership.

Worked example​

Ticket 39 fits numeric means/scales and categorical order on training rows, then writes them into the inert artifact. Ticket 40's batch command and API both load that state and call one prediction implementation. They never recompute a mean from the incoming batch and never fit categories on live requests.

The integration verifier sends these two raw instances through both public paths:

[
  {"prior_score": 48, "practice_minutes": 20, "learning_track": "foundations"},
  {"prior_score": 79, "practice_minutes": 68, "learning_track": "advanced"}
]

Both must return literal probabilities 0.015377 and 0.996779. This catches duplicated preprocessing that changes observable behavior; it does not prove that live source values are timely or semantically correct.

Historical correctness​

When building a training row for prediction time t, join each feature value that would have been available as of t, respecting source event time, ingestion delay, correction policy, and feature TTL. A latest-value join can leak future information. Labels usually mature later than features, so record their observation windows separately.

At serving time, choose what happens when a feature is missing, stale, late, or unavailable. Rejecting, using a documented default, reading a last-known value, or entering a degraded mode are product and risk decisions—not harmless data cleaning.

Detection​

Log or sample model version, feature-definition version, event/materialization timestamps, missingness, and safe distribution summaries under a privacy policy. When permitted, replay sampled serving features through offline computation and compare values and predictions. Monitor skew, but do not confuse a close aggregate distribution with row-level semantic parity.

Common mistakes​

  • Fitting normalization on each prediction batch.
  • Encoding a new category in a different order online.
  • Joining training data to the latest record instead of the as-of record.
  • Treating cached values as fresh because lookup latency is low.
  • Calling distribution similarity proof that individual features match.

Practice​

Define practice_minutes_last_7_days for Study Sprint. Specify event included/excluded rules, timezone, late-event corrections, prediction timestamp, historical join, online materialization, TTL, missing value, definition version, and a row-level parity test.

Next steps​

Shadow, Canary, and Blue–Green Rollouts uses observable compatibility and quality evidence to limit change risk.

Primary sources​

Checked 2026-10-15: Google Rules of ML on training–serving skew, TensorFlow Transform preprocessing guidance, Feast online-store semantics, and scikit-learn pipeline leakage guidance.