Skip to main content

Observational Product Analysis and Confounding

Product telemetry makes associations easy to calculate. It does not make treatment exchangeable. People who discover, adopt, or pay for a feature often differ in need, tenure, device, role, opportunity, and prior behavior from those who do not. Those differences can influence both exposure and outcome.

Prerequisites​

Complete decomposing metric movements and keep descriptive identities separate from causal explanations.

Learning goals​

  • State the target trial before interpreting an observational contrast.
  • Align eligibility, treatment assignment, time zero, follow-up, and outcome.
  • Identify plausible common causes without adjusting for mediators or colliders by reflex.
  • Route causal questions to a design capable of supporting them.

Decision contract​

The fixture compares four current-day users with four historical eligible actors. Current adoption is 4/4 after restatement; historical adoption is 2/4. The 50-point association is verified description. Calendar time, rollout eligibility, actor mix, delayed delivery, prior need, and simultaneous product changes remain possible explanations.

To ask whether the rollout caused adoption, specify: eligible actors before rollout; the strategy they could receive; assignment time; a fixed follow-up window; successful export as outcome; and the contrast of interest. Prefer randomized assignment where feasible. If observational estimation is necessary, justify the causal graph, measured confounders, positivity, consistency, missingness, and sensitivity analysis.

Do not control mechanically for post-treatment engagement, feature success, or support contact. Those can mediate effects or open biased paths. A predictive adjustment model is not automatically a causal model.

Worked example​

historical_eligible, historical_adopted = 4, 2
exposed, adopted = 4, 4
association = adopted / exposed - historical_adopted / historical_eligible
print(f"exposed={exposed} adopted={adopted} historical={historical_eligible} adopted_historical={historical_adopted}")
print(f"association_pp={association * 100:.1f} causal_effect=not_identified route=experimentation")

Expected output:

exposed=4 adopted=4 historical=4 adopted_historical=2
association_pp=50.0 causal_effect=not_identified route=experimentation

Limitations and common mistakes​

  • Before/after comparisons confound treatment with calendar time and other changes.
  • Comparing adopters with non-adopters conditions on a post-exposure behavior.
  • Regression adjustment cannot remove unmeasured confounding by assertion.
  • Instrumentation changes can mimic behavioral effects.
  • A narrow synthetic fixture cannot establish external validity or subgroup safety.

For experiment design, estimands, assignment checks, uncertainty, and quasi-experimental assumptions, continue in Experimentation & Causal Inference.

Practice​

Draw a causal graph with rollout eligibility, prior export need, device capability, exposure, successful use, and support contact. Mark which variables exist before assignment and explain which tempting adjustment could introduce bias.

Next steps​

Writing evidence-based decision memos communicates the verified association, competing hypotheses, and evidence needed next.

Primary sources​

Checked 2026-10-19: Hernán and Robins, Causal Inference: What If; CONSORT, CONSORT 2025 statement; NIST, Engineering Statistics Handbook.