Skip to main content

Metric Alerts and Anomaly Triage

An alert spends human attention. It should identify a decision-relevant symptom, name an owner, provide enough context for the first diagnostic step, and stop firing when no safe action exists. “Something looks unusual” is a query prompt, not an interrupt.

Prerequisites​

Complete dashboards as decision surfaces and retain its decision contract.

Learning goals​

  • Separate pipeline, definition, and behavioral anomalies.
  • Triage validity and close status before explaining behavior.
  • Treat static and adaptive thresholds as documented policies with backtests.
  • Record alert disposition so noise and missed cases can improve the system.

Decision contract​

  • Decision: Whether to quarantine data, hold rollout expansion, investigate behavior, or close the alert without product action.
  • Owner: data-platform owns event validity; core-product owns the rollout decision.
  • Threshold rationale: A duplicate-row rate above 0.04 catches the planted producer retry, while a 0.20 absolute adoption change catches the planted behavioral movement; neither threshold proves severity or cause.
  • Diagnostic path: Check missingness, schema, consent, duplicates, event and receive time, denominator, restatement, releases, segments, and guardrails—in that order.
  • Safe action: Deduplicate idempotently and hold expansion while a material data defect or guardrail breach is unresolved.
  • No-action condition: Close the product alert when validation shows only a metric-neutral duplicate; retain a data-quality ticket and its owner.

Triage order​

First ask whether the measurement system changed. A duplicate event ID can raise row counts while leaving a distinct-actor metric unchanged. Next ask whether the comparison is closed: the fixture's delayed review-017 event changes adoption from 3/4 at first close to 4/4 after restatement. Only then inspect product changes, population mix, and plausible mechanisms.

An anomaly is a verified departure from a declared expectation. A hypothesis is a candidate explanation. Root cause and causal effect require stronger evidence. Keep those labels separate in the incident record.

Worked example​

import json
from collections import Counter
from pathlib import Path

rows = [json.loads(line) for line in Path("tests/fixtures/product-analytics/review-events-v1.jsonl").read_text().splitlines()]
counts = Counter(row["event_id"] for row in rows)
duplicates = sorted(key for key, value in counts.items() if value > 1)
print(f"rows={len(rows)} unique_events={len(counts)} duplicate_ids={duplicates}")
print("data_quality=triage behavioral_change=investigate causal_claim=False")

Expected output:

rows=20 unique_events=19 duplicate_ids=['review-013']
data_quality=triage behavioral_change=investigate causal_claim=False

Limitations and common mistakes​

  • Alerting on every statistically unusual point creates noise and trains owners to ignore alerts.
  • A learned baseline can absorb past incidents, seasonality changes, or missing data; version it and backtest it.
  • Silence does not prove health when ingestion has stopped.
  • Automatic rollback may be unsafe when the signal is delayed, reversible, or outside the action's control.
  • Threshold changes made after seeing the incident destroy the audit trail unless versioned as a new policy.

Practice​

Write a runbook entry for each planted anomaly. Include evidence that would close it, an escalation timer, the smallest reversible action, and a field distinguishing “no impact” from “impact unknown.”

Next steps​

Feature adoption over the product lifecycle replaces one adoption total with stage-specific evidence.

Primary sources​

Checked 2026-10-19: Google SRE, Monitoring Distributed Systems and Tracking Outages; OpenTelemetry, Semantic conventions for events.