Skip to main content

Heterogeneous Treatment Effects

An average can conceal meaningful variation, but subgroup searches create many noisy opportunities for a compelling story. Heterogeneous treatment effects must be tied to pre-treatment variables, a decision, enough support in every arm, and uncertainty about the interaction.

The relevant question is not “is treatment significant for new students but not returning students?” It is “how different are the two treatment effects?” That requires an interaction contrast and its interval.

Prerequisites​

Complete Variance Reduction and CUPED and distinguish a conditional average treatment effect (CATE) from an individual treatment effect.

Learning goals​

  • Estimate subgroup effects using assigned arm within pre-treatment groups.
  • Test and report the interaction rather than comparing subgroup p-values.
  • Label analyses as pre-specified or exploratory and account for multiplicity and low power.

A pre-specified segment interaction​

Suppose segment was recorded before assignment and the new-versus-returning interaction was the only pre-specified effect modifier. Activation is observed for all students. The interval uses independent student-level Bernoulli variances and is only a large-sample teaching approximation.

import csv
import math
import statistics

with open("fixtures/experimentation-causal-inference/ab-test-users.csv", newline="") as handle:
    rows = list(csv.DictReader(handle))

effects, variances = {}, {}
for segment in ("new", "returning"):
    cells = {
        arm: [float(row["activated"]) for row in rows
              if row["segment"] == segment and row["assigned_arm"] == arm]
        for arm in ("control", "treatment")
    }
    effects[segment] = statistics.mean(cells["treatment"]) - statistics.mean(cells["control"])
    variances[segment] = sum(statistics.variance(cell) / len(cell) for cell in cells.values())

interaction = effects["new"] - effects["returning"]
interaction_se = math.sqrt(sum(variances.values()))
print(f"new-student effect={effects['new']:.3f}")
print(f"returning-student effect={effects['returning']:.3f}")
print(f"interaction={interaction:.3f} 95% CI=({interaction - 1.96 * interaction_se:.3f}, {interaction + 1.96 * interaction_se:.3f})")

The subgroup point estimates differ, but the interaction interval is broad. This fixture does not support a segment-specific rollout. Absence of strong interaction evidence is not proof that effects are identical; interaction tests often have far less power than the overall treatment comparison.

Pre-specified versus exploratory​

A confirmatory analysis names the modifier, cut points, outcome, scale, model, interaction, multiplicity family, and decision use before outcomes are available. Even pre-specifying dozens of subgroups leaves a multiplicity and power problem.

Exploratory trees, forests, or scans can generate useful hypotheses, but reuse of the same data for discovering and estimating favorable leaves produces optimism. Honest sample splitting or cross-fitting separates discovery from estimation, yet small leaves and many comparisons still demand caution. Replicate promising patterns in a new experiment.

Only pre-treatment characteristics can define interpretable baseline CATEs here. Subgroups formed by sessions, adherence, or outcome response are post-treatment selections. Also check overlap and sample size: a label with few treatment or control students cannot support a stable comparison, and a model cannot create missing support.

Common mistakes​

  • Comparing “significant” and “not significant” subgroup results instead of an interaction.
  • Searching many cuts and reporting only the most favorable leaf.
  • Calling a CATE an individualized causal effect.
  • Defining subgroups with post-treatment behavior.
  • Making targeted policy from wide, underpowered subgroup intervals.

Practice​

Write two labels for the segment analysis: one if it was the only preregistered interaction, and one if it was selected after scanning ten variables. Specify the multiplicity disclosure and replication step for each.

Next steps​

Project: Analyze an A/B Test combines design, estimands, metrics, missingness, guardrails, and uncertainty into one decision report.

Primary sources​

Checked 2026-10-15: Athey and Imbens, “Recursive Partitioning for Heterogeneous Causal Effects”, CONSORT 2025 explanation and elaboration, and Wang et al., “Statistics in Medicine—Reporting of Subgroup Analyses”.