Instrumentation Validation and Data Quality
Instrumentation quality is part of the measurement system. A query cannot recover an event that never fired, distinguish a retry without an idempotency key, or infer that a producer silently changed a trigger. Validation therefore spans collection, transport, schema, semantics, and metric output.
Prerequisites
Complete metric trees and the earlier tracking-plan lesson. Know the difference between a logical event and a delivered row.
Learning goals
- Apply deterministic checks in a safe order.
- Separate duplicates, late records, malformed properties, and consent violations.
- Monitor completeness and semantic drift rather than only schema validity.
- Quarantine invalid analytical input without erasing evidence of the incident.
Validation layers
- Envelope: required keys, parseable timestamps, and declared schema version.
- Taxonomy: known event name and purpose.
- Property schema: required fields with exact types and allowed values.
- Delivery: deduplicate by stable
event_id; measure retry and drop rates. - Time: compare
occurred_atwithreceived_at; apply the metric's cutoff and restatement policy. - Privacy: reject product-measurement events without the required consent or purpose authorization.
- Semantics: test that the event fires once at the documented business transition.
- Metric reconciliation: compare event-derived counts with independent operational records where permitted.
The fixture contains one duplicate delivery, one late account event, one malformed integer where feature must be a string, and one event collected across the consent boundary. These are different defects with different owners. A duplicate indicates delivery/idempotency trouble; a late event affects revision; a malformed property identifies schema drift; unauthorized measurement calls for privacy incident handling and collection repair.
At the first close, the validator keeps the earliest delivery per event_id, then quarantines three unique events: late, malformed, and unauthorized. Eleven unique records remain acceptable at that cutoff, including the essential consent_updated record. “Accepted by validation” does not mean every record belongs in every metric; each metric still applies its population and purpose contract.
Track rates and denominators, not only counts. A zero malformed count can mean a healthy producer or a producer that stopped sending anything. Segment health by producer version, platform, and schema version only when those properties are necessary and approved.
Worked example
import json
from datetime import datetime
from pathlib import Path
root = Path("tests/fixtures/product-analytics")
rows = [json.loads(line) for line in (root / "events-v1.jsonl").read_text().splitlines()]
unique = {}
for row in rows:
unique.setdefault(row["event_id"], row)
close = datetime.fromisoformat("2026-10-03T00:00:00+00:00")
late = [row for row in unique.values()
if datetime.fromisoformat(row["received_at"].replace("Z", "+00:00")) > close]
malformed = [row for row in unique.values()
if row["event_name"] == "feature_used" and not isinstance(row["properties"].get("feature"), str)]
unauthorized = [row for row in unique.values()
if row["purpose"] == "product_measurement" and not row["analytics_consent"]]
quarantined = {row["event_id"] for row in late + malformed + unauthorized}
accepted = [row for row in unique.values() if row["event_id"] not in quarantined]
print(f"duplicate_rows={len(rows)-len(unique)} late_rows_at_first_close={len(late)} malformed_rows={len(malformed)}")
print(f"unauthorized_measurement_rows={len(unauthorized)} accepted_first_close={len(accepted)} quarantined={len(quarantined)}")
Expected output:
duplicate_rows=1 late_rows_at_first_close=1 malformed_rows=1
unauthorized_measurement_rows=1 accepted_first_close=11 quarantined=3
Keeping categories separate makes remediation auditable. Silently dropping all four problems into a generic “bad rows” count would lose operational meaning.
Limitations and common mistakes
- Schema validity cannot prove correct trigger timing or complete coverage.
- Deduplicating on all columns fails when a retry changes only
received_at; use a logical idempotency key. - Dropping late data without reporting it biases results when lateness is systematic.
- Coercing malformed values can create false categories and hide producer regressions.
- Retaining quarantined personal data indefinitely creates another failure; define access, retention, and deletion procedures.
Practice
Design an alert for account_created completeness. Provide numerator, denominator, baseline window, minimum volume, owner, runbook, and a rule that avoids treating an empty pipeline as perfect quality.
Next steps
Apply validation and approved links without overclaiming identity in identity resolution and privacy boundaries.
Primary sources
Checked 2026-10-19: OpenTelemetry Semantic Conventions 1.44.0, event definitions and timestamps; JSON Schema, 2020-12 Core specification; W3C, Privacy Principles.