Skip to main content

Instrumentation Validation and Data Quality

Instrumentation quality is part of the measurement system. A query cannot recover an event that never fired, distinguish a retry without an idempotency key, or infer that a producer silently changed a trigger. Validation therefore spans collection, transport, schema, semantics, and metric output.

Prerequisites​

Complete metric trees and the earlier tracking-plan lesson. Know the difference between a logical event and a delivered row.

Learning goals​

  • Apply deterministic checks in a safe order.
  • Separate duplicates, late records, malformed properties, and consent violations.
  • Monitor completeness and semantic drift rather than only schema validity.
  • Quarantine invalid analytical input without erasing evidence of the incident.

Validation layers​

  1. Envelope: required keys, parseable timestamps, and declared schema version.
  2. Taxonomy: known event name and purpose.
  3. Property schema: required fields with exact types and allowed values.
  4. Delivery: deduplicate by stable event_id; measure retry and drop rates.
  5. Time: compare occurred_at with received_at; apply the metric's cutoff and restatement policy.
  6. Privacy: reject product-measurement events without the required consent or purpose authorization.
  7. Semantics: test that the event fires once at the documented business transition.
  8. Metric reconciliation: compare event-derived counts with independent operational records where permitted.

The fixture contains one duplicate delivery, one late account event, one malformed integer where feature must be a string, and one event collected across the consent boundary. These are different defects with different owners. A duplicate indicates delivery/idempotency trouble; a late event affects revision; a malformed property identifies schema drift; unauthorized measurement calls for privacy incident handling and collection repair.

At the first close, the validator keeps the earliest delivery per event_id, then quarantines three unique events: late, malformed, and unauthorized. Eleven unique records remain acceptable at that cutoff, including the essential consent_updated record. “Accepted by validation” does not mean every record belongs in every metric; each metric still applies its population and purpose contract.

Track rates and denominators, not only counts. A zero malformed count can mean a healthy producer or a producer that stopped sending anything. Segment health by producer version, platform, and schema version only when those properties are necessary and approved.

Worked example​

import json
from datetime import datetime
from pathlib import Path

root = Path("tests/fixtures/product-analytics")
rows = [json.loads(line) for line in (root / "events-v1.jsonl").read_text().splitlines()]
unique = {}
for row in rows:
unique.setdefault(row["event_id"], row)
close = datetime.fromisoformat("2026-10-03T00:00:00+00:00")
late = [row for row in unique.values()
if datetime.fromisoformat(row["received_at"].replace("Z", "+00:00")) > close]
malformed = [row for row in unique.values()
if row["event_name"] == "feature_used" and not isinstance(row["properties"].get("feature"), str)]
unauthorized = [row for row in unique.values()
if row["purpose"] == "product_measurement" and not row["analytics_consent"]]
quarantined = {row["event_id"] for row in late + malformed + unauthorized}
accepted = [row for row in unique.values() if row["event_id"] not in quarantined]
print(f"duplicate_rows={len(rows)-len(unique)} late_rows_at_first_close={len(late)} malformed_rows={len(malformed)}")
print(f"unauthorized_measurement_rows={len(unauthorized)} accepted_first_close={len(accepted)} quarantined={len(quarantined)}")

Expected output:

duplicate_rows=1 late_rows_at_first_close=1 malformed_rows=1
unauthorized_measurement_rows=1 accepted_first_close=11 quarantined=3

Keeping categories separate makes remediation auditable. Silently dropping all four problems into a generic “bad rows” count would lose operational meaning.

Limitations and common mistakes​

  • Schema validity cannot prove correct trigger timing or complete coverage.
  • Deduplicating on all columns fails when a retry changes only received_at; use a logical idempotency key.
  • Dropping late data without reporting it biases results when lateness is systematic.
  • Coercing malformed values can create false categories and hide producer regressions.
  • Retaining quarantined personal data indefinitely creates another failure; define access, retention, and deletion procedures.

Practice​

Design an alert for account_created completeness. Provide numerator, denominator, baseline window, minimum volume, owner, runbook, and a rule that avoids treating an empty pipeline as perfect quality.

Next steps​

Apply validation and approved links without overclaiming identity in identity resolution and privacy boundaries.

Primary sources​

Checked 2026-10-19: OpenTelemetry Semantic Conventions 1.44.0, event definitions and timestamps; JSON Schema, 2020-12 Core specification; W3C, Privacy Principles.