Skip to main content

Monitoring with Delayed Labels

Study Sprint completion is known only after the study period. At prediction time, service and input signals are available, but realized correctness is not. Later, labels may arrive for only a selected subset. Monitoring must not turn that subset into an imaginary representative population.

Prerequisites​

Complete Data Drift and Concept Drift. Understand event time, processing time, and the Study Sprint decision threshold.

Learning goals​

  • measure label delay, maturity, coverage, and selection separately.
  • join outcomes to the exact prediction and model version without leakage.
  • avoid claiming performance from an unrepresentative observed subset.

Worked example​

The delayed-selected-labels window has labels for 4 of 10 predictions. All four observed cases are classified correctly, so observed accuracy is 1.0. Coverage is only 0.4, and the observed cases were selected by the outcome process. The report therefore emits wait-for-representative-labels, not “promote” or “retrain,” and keeps continue-current-model-with-guardrails as its safe fallback.

Store a prediction record with event ID, entity or governed join key, event time, model and contract versions, score, thresholded decision, eligibility, and relevant slice fields. Store the outcome with its own event time, source, definition version, corrections, and observation time. Join point-in-time: never attach a later outcome or revised feature to the historical prediction as if it was known then.

Monitor a label-maturity curve: what fraction of eligible predictions has a valid outcome after 1, 7, or 30 days? Report performance only for a declared mature cohort, alongside coverage and censoring. Backfill corrected outcomes reproducibly rather than overwriting historical reports invisibly.

Selected labels arise when the decision controls whether the outcome is observed. If only learners admitted to a program produce completion labels, rejected learners are not missing at random. Accuracy on admitted learners does not identify accuracy for all applicants. Random audits, exploration, external outcomes, or causal designs may help, but they carry ethical and operational constraints.

Leading proxies—score distribution, overrides, short-term engagement—can be useful diagnostic clues. They are not substitutes for the target outcome unless that relationship has been validated and monitored.

Common mistakes​

  • Reporting preliminary accuracy without label coverage or cohort maturity.
  • Joining on a mutable user attribute instead of an immutable event identity.
  • Training on corrected labels while comparing with an unrevised reference.
  • Treating “not observed” as a negative outcome.
  • Ignoring that the model's own decision changes which labels become visible.

Practice​

For the Study Sprint fixture, define a seven-day and a thirty-day maturity cohort. Write what happens to late corrections, never-observed outcomes, withdrawals, and duplicate events. Explain why the 1.0 observed accuracy does not justify a deployment decision.

Next steps​

Continue with Production Performance and Calibration to evaluate mature labeled cohorts.

Primary sources​

Checked 2026-10-15: Selective Labels: Evaluating Predictions in the Presence of Unobservables, Google Rules of ML on feedback loops, NIST AI RMF Playbook, and W3C PROV data model.