Metric Alerts and Anomaly Triage
An alert spends human attention. It should identify a decision-relevant symptom, name an owner, provide enough context for the first diagnostic step, and stop firing when no safe action exists. “Something looks unusual” is a query prompt, not an interrupt.
Prerequisites
Complete dashboards as decision surfaces and retain its decision contract.
Learning goals
- Separate pipeline, definition, and behavioral anomalies.
- Triage validity and close status before explaining behavior.
- Treat static and adaptive thresholds as documented policies with backtests.
- Record alert disposition so noise and missed cases can improve the system.
Decision contract
- Decision: Whether to quarantine data, hold rollout expansion, investigate behavior, or close the alert without product action.
- Owner:
data-platformowns event validity;core-productowns the rollout decision. - Threshold rationale: A duplicate-row rate above 0.04 catches the planted producer retry, while a 0.20 absolute adoption change catches the planted behavioral movement; neither threshold proves severity or cause.
- Diagnostic path: Check missingness, schema, consent, duplicates, event and receive time, denominator, restatement, releases, segments, and guardrails—in that order.
- Safe action: Deduplicate idempotently and hold expansion while a material data defect or guardrail breach is unresolved.
- No-action condition: Close the product alert when validation shows only a metric-neutral duplicate; retain a data-quality ticket and its owner.
Triage order
First ask whether the measurement system changed. A duplicate event ID can raise row counts while leaving a distinct-actor metric unchanged. Next ask whether the comparison is closed: the fixture's delayed review-017 event changes adoption from 3/4 at first close to 4/4 after restatement. Only then inspect product changes, population mix, and plausible mechanisms.
An anomaly is a verified departure from a declared expectation. A hypothesis is a candidate explanation. Root cause and causal effect require stronger evidence. Keep those labels separate in the incident record.
Worked example
import json
from collections import Counter
from pathlib import Path
rows = [json.loads(line) for line in Path("tests/fixtures/product-analytics/review-events-v1.jsonl").read_text().splitlines()]
counts = Counter(row["event_id"] for row in rows)
duplicates = sorted(key for key, value in counts.items() if value > 1)
print(f"rows={len(rows)} unique_events={len(counts)} duplicate_ids={duplicates}")
print("data_quality=triage behavioral_change=investigate causal_claim=False")
Expected output:
rows=20 unique_events=19 duplicate_ids=['review-013']
data_quality=triage behavioral_change=investigate causal_claim=False
Limitations and common mistakes
- Alerting on every statistically unusual point creates noise and trains owners to ignore alerts.
- A learned baseline can absorb past incidents, seasonality changes, or missing data; version it and backtest it.
- Silence does not prove health when ingestion has stopped.
- Automatic rollback may be unsafe when the signal is delayed, reversible, or outside the action's control.
- Threshold changes made after seeing the incident destroy the audit trail unless versioned as a new policy.
Practice
Write a runbook entry for each planted anomaly. Include evidence that would close it, an escalation timer, the smallest reversible action, and a field distinguishing “no impact” from “impact unknown.”
Next steps
Feature adoption over the product lifecycle replaces one adoption total with stage-specific evidence.
Primary sources
Checked 2026-10-19: Google SRE, Monitoring Distributed Systems and Tracking Outages; OpenTelemetry, Semantic conventions for events.