Skip to main content

Regression Adjustment for Experiments

Randomization identifies the assignment effect; regression can improve precision. It should not redefine which students count, repair post-treatment selection, or turn an outcome-informed model search into a confirmatory result.

The estimand remains the all-randomized mean ITT score effect. This worked example uses the observed final scores and the pre-randomization baseline score. It therefore inherits the missing-outcome limitation from the unadjusted analysis.

Prerequisites​

Complete Difference in Means and Uncertainty and know which variables were measured before assignment.

Learning goals​

  • Distinguish precision adjustment from confounding control.
  • Use a prognostic pre-treatment covariate without conditioning on treatment response.
  • Report adjusted magnitude and uncertainty alongside the unadjusted result.

A prespecified baseline adjustment​

For a continuous outcome, one transparent working model includes assignment, centered baseline score, and—when the sample supports it—an assignment-by-baseline interaction. The analysis plan must state the covariates, transformations, interactions, variance estimator, and treatment of missing values before outcomes are inspected.

The compact calculation below estimates a common within-arm baseline slope, constructs adjusted outcomes, and compares their assignment-group means. Its unequal-variance normal interval is a teaching approximation; larger analyses should use the preregistered estimator and a design-compatible robust variance calculation.

import csv
import math
import statistics

with open("fixtures/experimentation-causal-inference/ab-test-users.csv", newline="") as handle:
    complete = [row for row in csv.DictReader(handle) if row["final_score"]]

groups = {
    arm: [row for row in complete if row["assigned_arm"] == arm]
    for arm in ("control", "treatment")
}
means = {
    arm: (
        statistics.mean(float(row["baseline_score"]) for row in group),
        statistics.mean(float(row["final_score"]) for row in group),
    )
    for arm, group in groups.items()
}
numerator = sum(
    (float(row["baseline_score"]) - means[arm][0])
    * (float(row["final_score"]) - means[arm][1])
    for arm, group in groups.items() for row in group
)
denominator = sum(
    (float(row["baseline_score"]) - means[arm][0]) ** 2
    for arm, group in groups.items() for row in group
)
theta = numerator / denominator
grand_baseline = statistics.mean(float(row["baseline_score"]) for row in complete)
adjusted = {
    arm: [
        float(row["final_score"])
        - theta * (float(row["baseline_score"]) - grand_baseline)
        for row in group
    ]
    for arm, group in groups.items()
}
effect = statistics.mean(adjusted["treatment"]) - statistics.mean(adjusted["control"])
se = math.sqrt(
    statistics.variance(adjusted["treatment"]) / len(adjusted["treatment"])
    + statistics.variance(adjusted["control"]) / len(adjusted["control"])
)
print(f"baseline slope={theta:.3f}")
print(f"adjusted effect={effect:.2f} score points")
print(f"95% interval=({effect - 1.96 * se:.2f}, {effect + 1.96 * se:.2f})")

Adjustment should be reported as a planned estimator, not as evidence that randomization “failed.” Comparing adjusted and unadjusted results is useful; choosing whichever crosses a threshold is not.

Never adjust for a consequence of treatment​

Sessions, questions answered, adherence, and satisfaction occur after assignment here. Conditioning on them can block part of the total effect, create selection through a collider, or target an undefined subset. A mediation or per-protocol question needs a new estimand and additional assumptions; it is not ordinary precision adjustment.

Covariates should be measured before assignment, prognostic of the outcome, available for nearly everyone, and chosen without inspecting treatment effects. Flexible or high-dimensional adjustment requires its own cross-fitting and inference plan. With small samples, model instability and chance extrapolation deserve explicit diagnostics.

Common mistakes​

  • Adding covariates until the treatment p-value becomes favorable.
  • Adjusting for sessions or another post-treatment variable.
  • Omitting blocks, pairs, clusters, or assignment probabilities from the analysis.
  • Reporting a model coefficient without defining its marginal estimand.
  • Assuming model adjustment fixes missing outcomes or noncompliance.

Practice​

Classify baseline score, assigned arm, sessions, received sprint, and final score by measurement time. Mark which can enter a prespecified precision adjustment for the total ITT effect and explain every exclusion.

Next steps​

Binary, Count, and Ratio Metrics changes the outcome scale while preserving student-level assignment.

Primary sources​

Checked 2026-10-15: Lin, “Agnostic Notes on Regression Adjustments to Experimental Data” and FDA, Adjusting for Covariates in Randomized Clinical Trials.