Difference in Means and Uncertainty

An experiment starts with an estimand, not a test. For the Study Sprint A/B test, the assignment unit is the student and the primary estimand is the intention-to-treat (ITT) difference in mean six-week score among all eligible randomized students, treatment assignment minus control assignment. Assignment is the intervention: receiving or using the sprint is not required to remain in the treatment group.

The fixed local fixture has 32 randomized students, 16 per arm. This lesson first estimates a complete-case contrast because four final scores are missing. That estimate is useful but does not, by itself, identify the all-randomized ITT estimand; Missing Data and Attrition will make the required assumptions explicit.

Prerequisites

Complete Preregistration and Analysis Plans. Keep its population, assignment unit, outcome horizon, and estimand fixed.

Learning goals

• Compute an assignment-group difference at the student randomization unit.
• Report an effect in score points and a confidence interval, not only a p-value.
• Separate sampling uncertainty from missing-outcome and design assumptions.

The unadjusted estimator

Let $Z_i$ be assigned arm and $Y_i$ the final score. The difference in assignment-group means is

$$\widehat{\tau}=\overline{Y}_{Z=1}-\overline{Y}_{Z=0}.$$

The following descriptive standard error treats randomized students as independent units and uses arm-specific sample variances. The normal interval is transparent for teaching, but this tiny fixture is not evidence that its large-sample approximation is accurate.

import csv
import math
import statistics

with open("fixtures/experimentation-causal-inference/ab-test-users.csv", newline="") as handle:
    rows = list(csv.DictReader(handle))

scores = {
    arm: [float(row["final_score"]) for row in rows
          if row["assigned_arm"] == arm and row["final_score"]]
    for arm in ("control", "treatment")
}
effect = statistics.mean(scores["treatment"]) - statistics.mean(scores["control"])
se = math.sqrt(
    statistics.variance(scores["treatment"]) / len(scores["treatment"])
    + statistics.variance(scores["control"]) / len(scores["control"])
)
low, high = effect - 1.96 * se, effect + 1.96 * se

print(f"observed students control={len(scores['control'])} treatment={len(scores['treatment'])}")
print(f"complete-case effect={effect:.2f} score points")
print(f"95% interval=({low:.2f}, {high:.2f})")

The point estimate is a practical magnitude on the preregistered scale. The interval describes uncertainty from observing finitely many independent assignment units under the stated approximation. It does not absorb outcome-definition changes, interference, broken randomization, measurement error, or assumptions about the four missing scores.

Design-consistent alternatives

A randomization interval can use the known assignment mechanism, while a model-based interval can use a prespecified outcome model. Blocking, unequal assignment probabilities, clustering, pairing, or repeated measurements change the estimator or variance calculation. Do not select a row-level test because it is convenient: reconstruct the actual assignment mechanism first.

With only 32 students, the interval will be wide. A wide interval is information about the design's precision, not permission to replace the planned outcome, drop inconvenient students, or interpret “not statistically significant” as proof of no important effect.

Common mistakes

• Treating sessions or questions as independent observations when students were randomized.
• Reporting only a p-value or a percentage lift without the score-point effect and interval.
• Switching from assigned arm to received treatment and silently changing the ITT estimand.
• Claiming the complete-case estimate is the all-randomized effect without a missing-data assumption.
• Reading every value inside a 95% interval as equally plausible or interpreting repeated-study coverage as a posterior probability.

Practice

Write a four-line result that names the population, assignment unit, estimand, point estimate, interval method, and the four missing outcomes. Then explain which uncertainty the interval omits.

Next steps

Regression Adjustment for Experiments uses prespecified pre-treatment information to improve precision without changing the assignment-based target.

Primary sources

Checked 2026-10-15: CONSORT 2025 explanation and elaboration and ICH E9(R1), estimands and sensitivity analysis.