Skip to main content

Data Quality in Production

Schema validation rejects malformed Study Sprint requests, but a valid number can still be wrong. A practice_minutes value of 60 might mean the last day offline and the last week online. Production data quality joins structural validation with semantic, temporal, and lineage checks.

Prerequisites​

Complete What to Monitor in ML Systems and recall the versioned inference contract from the Study Sprint service.

Learning goals​

  • separate schema validity from semantic truth and feature freshness.
  • choose checks at producer, transport, transformation, and consumer boundaries.
  • stop, quarantine, or degrade safely according to consequence.

Worked example​

In 2026-01-15/data-quality-incident, 7 invalid records and 3 missing-feature records occur among 99 non-service-error requests:

invalid_rate = (7 + 3) / (100 - 1) = 0.101010...

This is a manual boundary in the article because the fence records a result, not executable code. The monitoring verifier independently recomputes the same incident rate and decision from the repository-owned fixture.

The monitor emits halt-or-fallback, assigns data-product-on-call, links quarantine-invalid-input-and-trace-producer, and selects reject-or-use-approved-default-path. Accuracy remains 0.8 on the small labeled sample, but that does not clear the incident: rejected or silently defaulted cases may be absent from that denominator.

Check each boundary deliberately:

  • Schema: required fields, types, finite values, ranges, enums, and version compatibility.
  • Semantics: units, population, entity identity, impossible combinations, and whether zero differs from missing.
  • Freshness: event-time lag, source update time, watermark, and stale-cache age.
  • Completeness and volume: expected entities, missingness by field and segment, duplicates, and abrupt traffic loss.
  • Transformation parity: feature code/version, aggregation window, ordering, and offline–online comparison.
  • Lineage: producer, source snapshot, transformation revision, and consumers exposed.

Compare like with like. Day-of-week, release, geography, and eligibility changes can produce legitimate variation. Use distributions and slices in addition to averages because compensating errors can hide in a global mean.

Response design​

Fail closed when an invalid input could cause a high-consequence decision. A low-consequence workflow might return an explicit degraded response, defer the decision, or use an approved rule-based path. Imputation is a learned or governed policy; silently replacing malformed production values is not harmless resilience.

Preserve invalid counts and safe diagnostic context without logging secrets or unnecessary personal data. Trace the first bad producer version, determine exposure, correct the source, and replay only if idempotency and event-time semantics permit it.

Common mistakes​

  • Equating parse success with data quality.
  • Computing missingness only after rows were filtered out.
  • Alerting on every natural seasonal change.
  • Repairing consumer data while leaving the producer contract broken.
  • Using a default that changes eligibility without review.

Practice​

Add an impossible Study Sprint combination such as prior_score=100 with an undocumented sentinel for missingness. Decide where it should be rejected, which owner receives it, what users see, and which aggregate evidence is safe to retain.

Next steps​

Continue with Data Drift and Concept Drift to distinguish changes in inputs, outcomes, and relationships.

Primary sources​

Checked 2026-10-15: Google Rules of ML on training-serving skew and monitoring, TensorFlow Data Validation guide, NIST AI RMF Playbook, and W3C PROV overview.