Skip to main content

Alerts, Dashboards, and Runbooks

A metric becomes operational only when someone can interpret and act on it. The Study Sprint report encodes a monitor-to-runbook-to-decision path, including an explicit owner and safe fallback for every state.

Prerequisites​

Complete Slices, Fairness, and Harm Monitoring. Know the service, data, label, performance, and harm layers.

Learning goals​

  • distinguish pages, actionable tickets, dashboards, and analytical reports.
  • design thresholds using consequence, persistence, uncertainty, and error budget.
  • write a runbook that works under time pressure.

Worked example​

The service incident pages serving-on-call because error rate is 0.12 and p95 latency is 480 ms. Its runbook stabilizes capacity and dependencies; the safe fallback serves the last known good path or defers. Harmless covariate shift creates an investigation for model-operations-owner, but does not page or retrain. Delayed labels route to an audit of delay and selection.

Use the channel that matches urgency:

  • Page: immediate user impact where a trained responder can mitigate now.
  • Ticket: bounded investigation that can wait for working hours.
  • Dashboard: context, trends, segmentation, deployment annotations, and drill-down.
  • Report/review: mature outcome, fairness, and governance evidence requiring deliberation.

Alert on symptoms tied to user impact when possible. Combine magnitude, duration, traffic, data completeness, and multi-window persistence. Deduplicate related symptoms, suppress during declared maintenance, and annotate deployments and upstream changes. Test alerts and record time to detect, acknowledge, mitigate, and resolve.

A runbook should state purpose, severity, owner and backup, trigger definition, dashboards and queries, validation steps, immediate containment, approved fallback, rollback compatibility, exposure assessment, communications, escalation, recovery checks, evidence retention, and exit criteria. Commands must be safe to copy and should name the environment explicitly.

Alert fatigue is a reliability defect. Track false positives, unactionable pages, repeated alerts, and coverage gaps. Removing noise must preserve detection of meaningful harm; do not merely widen a threshold until the pager is quiet.

Common mistakes​

  • Paging on a noisy proxy that has no immediate action.
  • Showing percentages without volume, label coverage, or deployment annotations.
  • Assigning alerts to a team name with no current on-call owner.
  • Writing “investigate” instead of concrete validation and containment steps.
  • Automating retraining as the response to every drift alert.

Practice​

Write a complete Study Sprint alert for the data-quality incident. Include severity, exact denominator, two confirmation queries, owner, fallback, communication audience, recovery criteria, and what would make it a non-incident.

Next steps​

Continue with ML Incident Response to coordinate containment, investigation, recovery, and learning.

Primary sources​

Checked 2026-10-15: Google SRE monitoring distributed systems, Google SRE alerting on SLOs, Google SRE practical alerting, and NIST AI RMF Playbook.