Randomized vs. Observational Evidence
Randomized and observational studies can both contribute causal evidence, but they justify group comparability differently. Randomization controls the assignment mechanism by design. Observational analysis reconstructs a target contrast from treatment choices that occurred for other reasons.
Prerequisites
Complete Identification and Assumptions. Keep the target as the six-week ATE of assigning Study Sprint access versus usual resources.
Learning goals
- Explain how randomization and observational adjustment justify comparisons differently.
- Specify the target trial behind an observational analysis.
- Pair conclusions from either design with their distinct failure modes.
Randomized assignment
If eligible students are randomly assigned with known probabilities, assignment is independent of baseline potential outcomes in expectation. The difference in group means can therefore estimate the assignment effect without assuming that measured covariates include every common cause. Randomization also gives a known basis for uncertainty.
It does not guarantee an unbiased final report. Attrition, outcome missingness, treatment crossover, broken allocation, interference, selective analysis, ambiguous treatment versions, and measurement error can undermine the target or analysis. Chance imbalance can occur in a realized finite sample; prespecified baseline adjustment may improve precision but is not the source of exchangeability.
Observational assignment
In existing college records, motivated or prepared students may self-select into Study Sprint. Estimating an effect then requires a defensible account of treatment assignment, sufficient pre-treatment measurement, conditional exchangeability, positivity, consistency, and selection. Adjustment cannot recover comparisons that never occur or remove bias from an unmeasured common cause by declaration.
An observational study should specify the hypothetical target trial it seeks to emulate: eligibility, treatment strategies, assignment, start of follow-up, outcome, causal contrast, and analysis plan. Aligning eligibility, assignment, and time zero prevents avoidable biases but does not create randomization.
What each design can answer
| Feature | Randomized study | Observational study |
|---|---|---|
| Assignment comparability | Supplied by the random mechanism before post-assignment problems | Assumed after design and adjustment for sufficient causes |
| Typical strength | Clear internal comparison for the assigned intervention | Can study real-world populations, rare outcomes, or infeasible assignments |
| Central threats | Nonadherence, attrition, interference, protocol and measurement failures | Those threats plus confounding, selection, and limited overlap |
| Generalization | Requires target-population and implementation reasoning | Also requires transport reasoning; representativeness alone does not remove confounding |
Neither label creates universal evidence quality. A poorly executed randomized study may answer a narrow or distorted question; a carefully designed observational study may provide the best ethical and practical evidence while retaining explicit untestable assumptions.
Conclusions must carry assumptions
For a randomized offer: “Under correct allocation, well-defined access, adequate outcome follow-up, and the stated interference model, the mean difference estimates the assignment effect in enrolled eligible students.”
For self-selected enrollment: “Under consistency, sufficient control of pre-enrollment common causes, positivity, the stated interference model, valid measurement, and nonselective analysis, the standardized contrast identifies the target ATE.”
Common mistakes
- Calling a study an experiment because researchers compared two observed groups.
- Treating randomization as protection against post-randomization selection.
- Calling adjusted observational results “as good as randomized” because measured balance improved.
- Beginning follow-up after treatment-dependent survival or participation.
- Assuming a representative sample solves confounding or that randomization guarantees generalizability.
Practice
Write a target-trial specification for Study Sprint, then map each element to available observational records. Mark every mismatch as a design limitation rather than silently filling it with a model.
Next steps
Causal Questions from Domain Knowledge assembles the workflow into a decision-ready protocol.
Primary sources
Checked 2026-10-15: Rubin, “Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies”, Hernán and Robins, “Using Big Data to Emulate a Target Trial”, and Hernán and Robins, Causal Inference: What If.