Skip to main content

ML Incident Response

An ML incident can be an outage, corrupted feature, privacy breach, harmful decision pattern, invalid model version, or feedback loop. It may unfold slowly and affect only one segment. Incident response protects users first while preserving evidence for diagnosis.

Prerequisites​

Complete Alerts, Dashboards, and Runbooks. Know the Study Sprint registry, lineage, rollout, and fallback contracts.

Learning goals​

  • establish one incident owner with technical and risk-domain support.
  • contain harm before proving a root cause.
  • assess exposure across model, data, decision, time, and segment.

Worked example​

The concept-drift window has unchanged covariate bins but accuracy 0.2, Brier score 0.5005, calibration error 0.51, and conditional label delta 0.6. The path assigns ml-incident-commander, invokes contain-assess-exposure-and-rollback, and chooses previous-approved-model-or-human-review.

The incident owner coordinates rather than performing every task. Assign operations, data/model investigation, domain and harm assessment, privacy/security, communications, and scribe roles as needed. Keep a timestamped decision log with evidence and uncertainty.

Use a lifecycle:

  1. Declare and triage: describe observed impact, affected contract, severity, and unknowns.
  2. Contain: pause the use, reject bad input, route to human review, disable a feature, or rollback when compatible.
  3. Assess exposure: identify model versions, requests, decisions, users, slices, time range, and downstream consumers through forward lineage.
  4. Diagnose: test service, schema, freshness, training–serving parity, drift, labels, calibration, policy, and feedback hypotheses.
  5. Recover: validate the repair in shadow or bounded rollout, verify downstream state, monitor for recurrence, and communicate limitations.
  6. Learn and remedy: provide user remedy where appropriate; record contributing conditions, action owners, due dates, and verification.

Rollback is not universally safe. The previous model may share the bad upstream feature, violate the current schema, or encode the same harm. Confirm artifact integrity, compatibility, approvals, feature availability, and capacity. Some decisions are irreversible; recovery must include affected-user and downstream-data remediation.

Common mistakes​

  • Waiting for root-cause certainty before containing active harm.
  • Having several coordinators but no incident owner.
  • Rolling back the model while leaving corrupted features in place.
  • Measuring exposure only by request count instead of decisions and consequences.
  • Writing a blame-focused postmortem without verified corrective actions.

Practice​

Create a timeline for the Study Sprint concept-drift incident. Include first bad decision, detection, declaration, containment, exposure query, rollback decision, recovery validation, communication, and two corrective actions with owners.

Next steps​

Continue with Retraining Policies and Feedback Loops to decide when changing the model is warranted.

Primary sources​

Checked 2026-10-15: NIST AI RMF 1.0, NIST AI RMF Playbook Manage function, Google SRE incident response, and Google SRE postmortem culture.