Project: Analyze an A/B Test

This project turns the fixed Study Sprint fixture into a concise decision report. The task is not to find a favorable p-value. It is to preserve the randomized design, estimate the preregistered effects, show uncertainty and data loss, inspect guardrails, and state what the evidence does and does not justify.

Prerequisites

Complete Heterogeneous Treatment Effects and keep the preregistered plan from the design batch beside you.

Learning goals

• Reproduce a user-level A/B-test analysis from a local immutable fixture.
• Lead with the estimand, magnitude, and interval before decision language.
• Integrate missingness, noncompliance, multiplicity, guardrails, and exploratory subgroups.

Frozen analysis contract

Before opening outcomes, record this contract:

Field	Decision
Population	32 eligible randomized students in the fixture
Assignment unit	Student (user_id)
Treatment	Offer of the Study Sprint package
Primary estimand	All-randomized ITT difference in mean six-week score
Primary estimator	Assignment-group mean difference; complete-case result plus missing-score sensitivity
Precision analysis	Prespecified baseline-score adjustment, reported with the raw estimate
Secondary family	Activation and sessions per randomized student; Holm adjustment if confirmatory
Guardrail	Difference in reported adverse-event risk
Heterogeneity	New-versus-returning interaction; exploratory unless preregistered
Decision	Weigh worthwhile magnitude, uncertainty, harms, missingness, feasibility, and replication—not a binary significance rule

Reproduce the report

The script below uses only user-level rows. The normal intervals are transparent teaching approximations for this small fixture, not a production inference engine.

import csv
import math
import statistics

with open("fixtures/experimentation-causal-inference/ab-test-users.csv", newline="") as handle:
    rows = list(csv.DictReader(handle))
groups = {arm: [row for row in rows if row["assigned_arm"] == arm]
          for arm in ("control", "treatment")}

def difference(field, observed=lambda row: True):
    values = {arm: [float(row[field]) for row in group if observed(row)]
              for arm, group in groups.items()}
    effect = statistics.mean(values["treatment"]) - statistics.mean(values["control"])
    se = math.sqrt(sum(statistics.variance(v) / len(v) for v in values.values()))
    return effect, effect - 1.96 * se, effect + 1.96 * se

score = difference("final_score", lambda row: bool(row["final_score"]))
activation = difference("activated")
guardrail = difference("guardrail_event")
missing = {arm: sum(not row["final_score"] for row in group) for arm, group in groups.items()}

print("ESTIMAND: ITT mean six-week score effect; student assignment unit")
print(f"PRIMARY: {score[0]:.2f} points; 95% CI ({score[1]:.2f}, {score[2]:.2f})")
print(f"SECONDARY: activation risk difference {activation[0]:.3f}; 95% CI ({activation[1]:.3f}, {activation[2]:.3f})")
print(f"GUARDRAIL: adverse-event risk difference {guardrail[0]:.3f}; 95% CI ({guardrail[1]:.3f}, {guardrail[2]:.3f})")
print(f"MISSING: control={missing['control']}/16 treatment={missing['treatment']}/16")
print("DECISION: do not ship from a significance threshold; resolve missing-score and harm uncertainty.")

Interpret, do not vote

The complete-case score estimate is above the preregistered four-point worthwhile effect, but its interval includes effects below that threshold and the all-randomized result depends on missing-score assumptions. Baseline adjustment is much more precise in this synthetic fixture, yet it inherits the same missing-outcome limitation. The activation interval is wide. The treatment arm also has more recorded guardrail events, with substantial uncertainty.

A defensible decision might continue evidence collection, investigate every guardrail event and missing score, verify instrumentation, and decide whether a larger preregistered replication is proportionate. It would not say “ship because adjusted p is below 0.05” or “cancel because the unadjusted interval crosses zero.” Statistical uncertainty is one input alongside practical value, harm, implementation fidelity, cost, fairness, and reversibility.

Required report appendix

Attach the fixture version, analysis code, software version, allocation and observation counts, outcome definitions, raw arm summaries, effect scales, interval methods, multiplicity family, missing-data scenarios, receipt first stage, all guardrails, deviations, and whether the segment interaction was pre-specified or exploratory. Preserve null and unfavorable results.

Add checks for duplicate user IDs, valid arm labels, expected 16:16 assignment counts, impossible values, outcome-window violations, and sample-ratio mismatch. A clean check does not prove the causal assumptions, but a failed check can invalidate the analysis.

Common mistakes

• Starting with dashboard significance instead of the estimand and assignment unit.
• Treating sessions as independent rows or receipt as randomized.
• Omitting missing students and adverse events from the executive summary.
• Turning the strongest subgroup into the product target after exploration.
• Recommending “ship” solely because one interval excludes zero.

Practice

Write a six-sentence memo: decision, estimand, primary magnitude and interval, guardrail result, missingness sensitivity, and recommended next action. A reviewer should be able to disagree with the decision without disputing what was estimated.

Next steps

Continue to Matching and Balance, where assignment is not randomized and identification requires stronger assumptions and falsification checks.

Primary sources

Checked 2026-10-15: CONSORT 2025 explanation and elaboration, ICH E9(R1), estimands and sensitivity analysis, and Kohavi et al., “A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls”.