Variance Reduction and CUPED

CUPED—Controlled-experiment Using Pre-Experiment Data—subtracts predictable outcome variation using a covariate measured before exposure. When planned and implemented correctly, it can narrow uncertainty without changing the randomized ITT estimand.

It cannot create information about missing outcomes, correct a broken assignment, or legitimize choosing covariates after seeing treatment effects. The covariate must be unaffected by treatment; a concurrent session count is not “almost pre-treatment.”

Prerequisites

Complete Noncompliance, ITT, and Treatment on the Treated and review baseline adjustment.

Learning goals

• Construct a centered CUPED outcome from pre-experiment data.
• Quantify precision gain separately from effect magnitude.
• Recognize leakage, drift, and missing-covariate failure modes.

Adjustment and variance reduction

For outcome $Y$, pre-experiment covariate $X$, and frozen coefficient $\theta$, define

$$Y_i^{\text{cuped}}=Y_i-\theta(X_i-\overline X).$$

Centering preserves the overall scale. Here the within-arm slope is estimated only to demonstrate the arithmetic; production plans should state whether $\theta$ comes from historical data, blinded pooled data, sample splitting, or the experiment, and use matching inference.

import csv
import math
import statistics

with open("fixtures/experimentation-causal-inference/ab-test-users.csv", newline="") as handle:
    rows = [row for row in csv.DictReader(handle) if row["final_score"]]
groups = {arm: [row for row in rows if row["assigned_arm"] == arm]
          for arm in ("control", "treatment")}
xbar = {arm: statistics.mean(float(r["baseline_score"]) for r in group)
        for arm, group in groups.items()}
ybar = {arm: statistics.mean(float(r["final_score"]) for r in group)
        for arm, group in groups.items()}
theta = sum(
    (float(r["baseline_score"]) - xbar[arm]) * (float(r["final_score"]) - ybar[arm])
    for arm, group in groups.items() for r in group
) / sum(
    (float(r["baseline_score"]) - xbar[arm]) ** 2
    for arm, group in groups.items() for r in group
)
grand_x = statistics.mean(float(r["baseline_score"]) for r in rows)
raw = {arm: [float(r["final_score"]) for r in group] for arm, group in groups.items()}
cuped = {arm: [float(r["final_score"]) - theta * (float(r["baseline_score"]) - grand_x)
               for r in group] for arm, group in groups.items()}

def estimate(values):
    effect = statistics.mean(values["treatment"]) - statistics.mean(values["control"])
    se = math.sqrt(sum(statistics.variance(v) / len(v) for v in values.values()))
    return effect, se

raw_effect, raw_se = estimate(raw)
cuped_effect, cuped_se = estimate(cuped)
reduction = 1 - cuped_se ** 2 / raw_se ** 2
print(f"raw effect={raw_effect:.2f} SE={raw_se:.2f}")
print(f"CUPED effect={cuped_effect:.2f} SE={cuped_se:.2f}")
print(f"estimated variance reduction={reduction:.1%}")

The synthetic fixture was deliberately built with an unusually predictive baseline, so its variance reduction is not a benchmark. Real gains depend on stable correlation, covariate coverage, outcome definition, and population. Report the observed correlation and precision gain without promising the same percentage elsewhere.

The adjusted effect can differ from the raw effect in a finite realized sample because treatment and control have slightly different observed baseline means. That is compatible with targeting the same population effect. Report both estimates and follow the preregistered primary estimator rather than selecting the more favorable one.

Operational checks

Freeze the pre-period before assignment, version the metric code, and verify that treatment cannot alter late-arriving “pre” data. Handle new users and missing histories prospectively. Re-estimate or validate $\theta$ when metric definitions or populations drift. Preserve student-level randomization in the variance estimator.

Common mistakes

• Using post-treatment engagement as the CUPED covariate.
• Fitting many covariates and retaining the one with the best treatment result.
• Treating variance reduction as bias correction.
• Comparing CUPED and raw p-values and choosing the smaller one.
• Advertising synthetic-fixture precision gains as a production guarantee.

Practice

Write a data-lineage check for baseline score: measurement window, freeze time, late-event policy, missingness rule, version, and proof that assignment cannot affect it.

Next steps

Heterogeneous Treatment Effects asks whether effects differ across pre-treatment groups.

Primary sources

Checked 2026-10-15: Deng et al., “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data”, Lin, “Agnostic Notes on Regression Adjustments to Experimental Data”, and FDA, Adjusting for Covariates in Randomized Clinical Trials.