Skip to main content

Matching and Balance

Matching asks a design question before comparing outcomes: which untreated units could credibly represent what would have happened to the treated units? It can make comparisons less dependent on distant extrapolation. It cannot recover an unmeasured common cause or turn voluntary treatment into random assignment.

Prerequisites​

Complete Project: Analyze an A/B Test and review Identification and Assumptions. Here treatment is chosen rather than randomized.

Learning goals​

  • Tie a match to an assignment mechanism, target estimand, and pre-treatment covariates.
  • assess balance in distributions and overlap after matching.
  • use failed balance and falsification checks to reject a design without calling passed checks proof.

Design before outcomes​

Suppose students voluntarily enroll in Study Sprint. The target is the ATT: the mean six-week score effect for students who enrolled, comparing enrollment with no enrollment. A plausible assignment account says prior score and course load influence enrollment and the outcome. Match on those pre-treatment causes; do not match on attendance, completed exercises, or other consequences of enrollment.

Identification still requires consistency, no relevant interference, overlap for treated students, accurate measurement, and conditional exchangeability: after the chosen baseline variables, enrollment is as-if independent of both potential outcomes. Matching addresses only measured variables in that claim.

Worked match and matched failure check​

This deterministic simulation is a teaching fixture, not empirical validation. Each treated student is paired to the nearest untreated student on baseline score. The prior-outcome negative control is deliberately imbalanced, revealing that the apparently good baseline-score match missed prior motivation.

import math
import statistics

treated = [(50, 62, 54), (60, 70, 64), (70, 83, 74), (80, 90, 84)]
controls = [(49, 52, 50), (59, 60, 60), (69, 73, 70), (79, 80, 80), (30, 50, 30), (95, 88, 94)]
available = controls.copy()
pairs = []
for baseline, outcome, prior in treated:
    control = min(available, key=lambda row: abs(row[0] - baseline))
    available.remove(control)
    pairs.append(((baseline, outcome, prior), control))

raw = statistics.mean(row[1] for row in treated) - statistics.mean(row[1] for row in controls)
matched = statistics.mean(t[1] - c[1] for t, c in pairs)
prior_gap = statistics.mean(t[2] - c[2] for t, c in pairs)
baseline_gap = statistics.mean(t[0] - c[0] for t, c in pairs)
print(f"raw outcome difference={raw:.2f}")
print(f"matched ATT estimate={matched:.2f}")
print(f"matched baseline-score gap={baseline_gap:.2f}")
print(f"negative-control prior-outcome gap={prior_gap:.2f}")

The matched estimate is a descriptive contrast unless the identification assumptions hold. The small baseline-score gap shows what the algorithm achieved on that variable; it does not test unmeasured exchangeability. The four-point prior-outcome gap is a falsification warning: redesign with a credible pre-treatment motivation measure, seek another design, or conclude that the ATT is not identified from these data.

Diagnose the matched design​

Inspect standardized mean differences, variances, quantiles, and plots for every baseline covariate, interactions, and important transformations. Show the unmatched and matched samples, discarded units, reuse of controls, distances, and effective target population. A conventional absolute standardized-difference threshold is a descriptive flag, not a causal certificate.

Check overlap before selecting an algorithm. A treated unit with no comparable control defines an unsupported region; dropping it changes the estimand from the original ATT to an overlap-restricted population. Exact or coarsened matching can protect decisive categories, while calipers can reject bad pairs. Report unmatched treated units rather than quietly forcing matches.

Useful falsification checks include outcomes measured before treatment, exposures that treatment cannot affect, alternative timing, and deliberately omitted outcome data during design. Passing them can fail to reveal bias; failing them can reveal a broken design.

Failure modes​

  • An unmeasured motivation variable affects both enrollment and scores.
  • Matching on post-treatment engagement blocks part of the effect and may open selection paths.
  • Poor overlap forces distant pairs or silently changes the population.
  • Interference occurs when matched students share Sprint materials.
  • Repeatedly tuning the match after viewing outcomes makes design choices outcome-dependent.

Common mistakes​

  • Calling a low balance statistic evidence that treatment was randomized.
  • Matching on every available column without a causal timing argument.
  • reporting only the propensity score rather than covariate distributions.
  • ignoring discarded treated units and the changed estimand.
  • treating nearest-neighbor matching as automatically unbiased or efficient.

Practice​

Draw a DAG for voluntary enrollment, prior score, motivation, course load, peer sharing, and final score. Mark which variables are eligible for matching. Write an ATT conclusion template that explicitly conditions on measured exchangeability and states what a failed prior-outcome check would change.

Next steps​

Propensity Scores and Weighting replaces explicit pairs with a modeled assignment probability and makes overlap and influential weights central diagnostics.

Primary sources​

Checked 2026-10-15: Rosenbaum and Rubin, “The Central Role of the Propensity Score”, Stuart, “Matching Methods for Causal Inference”, and Austin, balance diagnostics.