Skip to main content

Noncompliance, ITT, and Treatment on the Treated

Assignment and receipt answer different causal questions. The intention-to-treat effect compares students by randomized offer, regardless of whether they received Study Sprint. It preserves the randomization and estimates the policy effect of offering the program under observed uptake.

A raw comparison of recipients with nonrecipients is not a treatment-on-the-treated effect: receipt is a post-assignment choice related to motivation, access, and prognosis. Randomization does not make that self-selected comparison exchangeable.

Prerequisites​

Complete Missing Data and Attrition and distinguish assignment, delivery, use, and measurement.

Learning goals​

  • Report ITT by original assigned arm.
  • Quantify how strongly assignment changed receipt.
  • Interpret a Wald instrumental-variable ratio as a complier effect only under explicit assumptions.

Offer effect and complier effect​

Activation is observed for all 32 students. The ITT risk difference is therefore a clean assignment-group contrast for this fixture. Under relevance, random assignment, exclusion, monotonicity, and a well-defined receipt version, dividing that ITT effect by the assignment effect on receipt identifies a local average treatment effect for compliers—not the effect for everyone who happened to receive treatment.

import csv
import math
import random
import statistics

with open("fixtures/experimentation-causal-inference/ab-test-users.csv", newline="") as handle:
    rows = list(csv.DictReader(handle))
groups = {arm: [row for row in rows if row["assigned_arm"] == arm]
          for arm in ("control", "treatment")}

def contrast(sample, field):
    return (
        statistics.mean(float(row[field]) for row in sample["treatment"])
        - statistics.mean(float(row[field]) for row in sample["control"])
    )

itt = contrast(groups, "activated")
first_stage = contrast(groups, "received_sprint")
cace = itt / first_stage
se_itt = math.sqrt(sum(
    statistics.variance(float(row["activated"]) for row in group) / len(group)
    for group in groups.values()
))
rng = random.Random(3706)
boot = []
for _ in range(10000):
    sample = {arm: rng.choices(group, k=len(group)) for arm, group in groups.items()}
    stage = contrast(sample, "received_sprint")
    if abs(stage) > 1e-12:
        boot.append(contrast(sample, "activated") / stage)
boot.sort()
low, high = boot[int(0.025 * (len(boot) - 1))], boot[int(0.975 * (len(boot) - 1))]

print(f"activation ITT={itt:.3f} 95% CI=({itt - 1.96 * se_itt:.3f}, {itt + 1.96 * se_itt:.3f})")
print(f"receipt first stage={first_stage:.3f}")
print(f"complier effect={cace:.3f} bootstrap 95% interval=({low:.3f}, {high:.3f})")

The bootstrap resamples students within assigned arms, so the uncertainty unit matches randomization. With only 32 students, its percentile interval is illustrative and unstable; weak first stages can make ratio estimates explosive, and a stronger design or randomization-aware method is needed for serious inference.

The exclusion restriction says assignment affects activation only through the defined receipt. Monotonicity rules out students who would receive the sprint only when assigned control. The result is local to compliers under this assignment mechanism. Spillovers, multiple treatment versions, or direct encouragement effects can invalidate that interpretation.

“Treatment on the treated” is often used loosely. Name the actual estimand—such as the complier average causal effect—and the population it covers. Always keep the ITT result because it is identified by the randomized offer under fewer behavioral assumptions and often matches the operational decision.

Common mistakes​

  • Dropping noncompliers from their assigned arms.
  • Comparing recipients with nonrecipients as if receipt were randomized.
  • Calling the Wald ratio an ATE for all students.
  • Ignoring a weak or heterogeneous first stage.
  • Assuming exclusion and monotonicity follow automatically from randomization.

Practice​

Draw the four principal receipt types implied by potential receipt under each assignment. State which types are ruled out, which population the ratio targets, and how a direct reminder effect would violate exclusion.

Next steps​

Variance Reduction and CUPED returns to prespecified pre-experiment information for greater precision.

Primary sources​

Checked 2026-10-15: Angrist, Imbens, and Rubin, “Identification of Causal Effects Using Instrumental Variables”, ICH E9(R1), estimands and sensitivity analysis, and CONSORT 2025 explanation and elaboration.