Skip to main content

Missing Data and Attrition

Randomization balances assignment at baseline; it does not make post-assignment missing outcomes harmless. The all-randomized ITT estimand still concerns 32 students, while the primary score is observed for only 28. A complete-case contrast targets the ITT effect only under assumptions about why scores are missing.

Prerequisites​

Complete Multiple Metrics and Multiple Testing and retain the original randomized denominators.

Learning goals​

  • Separate an intercurrent event from an unobserved outcome.
  • Report attrition counts, reasons, timing, and baseline patterns by assigned arm.
  • Use sensitivity analysis to show how untestable assumptions affect the decision.

Diagnose before modeling​

Create a participant flow from eligibility through assignment, delivery, follow-up, and analysis. The fixture records two missing scores per arm, but equal rates do not prove equal missingness mechanisms. lost_contact, schedule_conflict, and technical_failure may imply different relationships with the unobserved score.

Continue collecting the planned outcome after treatment discontinuation when the estimand requires it. A student who did not receive the sprint can still have an observed score; noncompliance is not automatically missing data.

A transparent sensitivity range​

The following exercise declares a plausible missing-score gain range of 0 to 10 points. Its adverse scenario assigns 0-point gains to missing treatment students and 10-point gains to missing control students; the favorable scenario reverses those assignments. These are not distribution-free bounds—scores outside this gain range would widen them.

import csv
import statistics

with open("fixtures/experimentation-causal-inference/ab-test-users.csv", newline="") as handle:
    rows = list(csv.DictReader(handle))
groups = {arm: [row for row in rows if row["assigned_arm"] == arm]
          for arm in ("control", "treatment")}
observed = {arm: [float(row["final_score"]) for row in group if row["final_score"]]
            for arm, group in groups.items()}
complete_case = statistics.mean(observed["treatment"]) - statistics.mean(observed["control"])

def scenario(treatment_gain, control_gain):
    values = {}
    for arm, group in groups.items():
        gain = treatment_gain if arm == "treatment" else control_gain
        values[arm] = [
            float(row["final_score"]) if row["final_score"]
            else float(row["baseline_score"]) + gain
            for row in group
        ]
    return statistics.mean(values["treatment"]) - statistics.mean(values["control"])

print("missing control=2/16 treatment=2/16")
print(f"complete-case effect={complete_case:.2f}")
print(f"declared sensitivity range=({scenario(0, 10):.2f}, {scenario(10, 0):.2f})")

Multiple imputation can propagate uncertainty under a specified missing-at-random model; inverse-probability weighting models observation; likelihood methods make their own assumptions. None can recover unobserved outcomes from data alone when missingness depends on those outcomes after conditioning. Pair a primary method with credible departures, such as delta adjustments or pattern-mixture scenarios, and report whether the decision changes.

Do not use “last observation carried forward” as a neutral default. Do not impute within treatment groups and then treat filled values as known. Imputation and observation models should respect assigned arm, prognostic baselines, design strata, interactions needed by the estimand, and uncertainty from estimating missing values.

Common mistakes​

  • Calling equal attrition rates proof of no attrition bias.
  • Deleting missing rows and still saying “all randomized students.”
  • Treating discontinuation, nonadherence, and missing measurement as synonyms.
  • Reporting one favorable missing-at-random model without sensitivity analysis.
  • Omitting reasons and denominators by assigned arm.

Practice​

Draw the 32-student flow and list which assumption makes the complete-case estimate valid. Then propose two outcome-informed departures that are plausible for each recorded attrition reason.

Next steps​

Noncompliance, ITT, and Treatment on the Treated keeps assignment effects separate from receipt effects.

Primary sources​

Checked 2026-10-15: National Research Council, The Prevention and Treatment of Missing Data in Clinical Trials, ICH E9(R1), estimands and sensitivity analysis, and CONSORT 2010 explanation and elaboration.