What to Monitor in ML Systems
A deployed model is one component in a decision system. Monitoring only its accuracy misses outages; monitoring only CPU and latency misses silently harmful decisions. The Study Sprint classifier therefore needs a layered contract that joins operational signals to the model card's intended use, exclusions, evaluation slices, threshold, and owner.
Prerequisites
Complete Project: Serve a scikit-learn Model. Keep its immutable model version, request schema, lineage, rollout, and fallback contracts available.
Learning goals
- distinguish service health, data quality, drift, model performance, decisions, and user outcomes.
- assign every signal a reference, window, owner, response, and limitation.
- treat proxy signals as diagnostic clues rather than proof of user harm.
Worked example
Run the local time-indexed simulation from the repository root:
python examples/ml-engineering/monitor.py \
--input fixtures/ml-engineering/monitoring-windows.json \
--output build/study-sprint/monitoring-report.json
The report keeps six layers separate:
| Layer | Study Sprint signal | Question it answers |
|---|---|---|
| service health | error rate and p95 latency | Can eligible requests receive a result on time? |
| data quality | invalid and missing-field rate | Does input satisfy the serving contract? |
| covariate and label shift | feature-bin and outcome-prevalence change | Has the observed population changed? |
| concept drift | change in observed label rate within score bands | Has the input–outcome relationship changed? |
| performance drift | accuracy, Brier score, calibration error | Has realized predictive behavior changed on labeled cases? |
| outcomes and harms | slice metrics, overrides, complaints, adverse events | Who experiences what consequence? |
Availability does not establish correctness. Data drift does not establish concept drift. Aggregate performance does not establish safety for every segment. A complaint count may be affected by access and reporting. Each signal requires context.
The training and serving contract supplies model version, schema, feature definitions, threshold, intended use, evaluation reference, and fallback. The model card adds excluded uses, known limitations, slice evidence, and accountable owners. Without these anchors, a dashboard can be precise but uninterpretable.
Monitoring contract
For every metric, record its numerator and denominator, event time, processing time, cohort, reference distribution, expected label delay, missingness, threshold rationale, escalation owner, runbook, and retention policy. Version that contract beside the deployment. Changes to a threshold are governed changes, not dashboard cosmetics.
Start from failure modes and consequences. Page the serving owner for a severe availability breach; create an investigation for drift with healthy outcomes; route a suspected segment harm to the model-risk owner. No single divergence threshold should automatically mean retrain.
Common mistakes
- Calling all changes “model drift” and losing the failure boundary.
- Comparing production traffic with a convenient but irrelevant training average.
- Omitting denominators, label coverage, model version, or decision exposure.
- Logging sensitive raw features when aggregates would answer the question.
- Treating a green dashboard as evidence that no unmeasured harm exists.
Practice
Write a monitoring contract for one Study Sprint decision. Name one leading operational signal, one lagging outcome, the cohort and window for each, their owners, and a safe fallback. State one failure neither signal can detect.
Next steps
Continue with Data Quality in Production to monitor the inputs before interpreting model metrics.
Primary sources
Checked 2026-10-15: NIST AI Risk Management Framework 1.0, NIST AI RMF Playbook, Google Rules of ML, Google SRE monitoring distributed systems, and Model Cards for Model Reporting.