Skip to main content

Sensitivity Analysis and Honest Conclusions

Sensitivity analysis asks how conclusions change under explicit departures from assumptions. It does not assign a probability that assumptions are true, erase bias, or convert a fragile observational design into an experiment. An honest analysis can end with “not identified from these data.”

Prerequisites​

Complete Synthetic Controls and the full causal learning path. Keep the decision, estimand, assignment mechanism, and assumption ledger visible.

Learning goals​

  • pair each identification assumption with diagnostics, falsification tests, and a quantitative stress test where possible.
  • benchmark hidden-bias scenarios against measured causes without claiming equivalence.
  • write conclusions that preserve population, treatment, outcome, uncertainty, assumptions, and unresolved threats.

An assumption-to-action ledger​

For matching or weighting, vary the strength of an omitted common cause and examine negative controls. For DiD and event studies, bound departures from parallel trends and anticipation. For RD, vary bandwidths and probe manipulation and other cutoff rules. For IV, quantify direct-effect or monotonicity violations. For synthetic control, vary donors, timing, fit windows, and spillovers.

Sensitivity must target the claimed estimand. A robust ATT among overlap-supported enrollees does not establish a population ATE. Every restriction, trimming rule, bandwidth, donor exclusion, or complier interpretation may change the target.

Worked tipping point and failed falsification​

This deterministic simulation starts with a four-point observational contrast. A bias parameter subtracts up to five points. The conclusion crosses zero at four points of hidden bias, while a two-point negative-control association already shows residual structure.

observed_effect = 4.0
hidden_bias_scenarios = range(0, 6)
adjusted = {bias: observed_effect - bias for bias in hidden_bias_scenarios}
tipping_point = min(bias for bias, effect in adjusted.items() if effect <= 0)
negative_control_association = 2.0
print("bias-adjusted effects=" + ", ".join(f"{b}:{e:.1f}" for b, e in adjusted.items()))
print(f"zero-effect tipping point={tipping_point} points of bias")
print(f"negative-control association={negative_control_association:.1f}")
print("CONCLUSION: causal effect not identified from these data")

This is not a universal sensitivity method; it teaches a tipping-point report. A substantive analysis would map a bias model to the estimator and benchmark it against credible measured variables. The negative-control failure is direct evidence against the current design, so the correct response is redesign or a noncausal conclusion—not selecting a more favorable specification.

Diagnostics are asymmetric​

Falsification tests can contradict implications of a design. Passing them only says the specific contradiction was not observed with available power and measurement. Thus:

  • balance can reveal residual measured differences but cannot rule out hidden bias;
  • propensity prediction cannot validate exchangeability;
  • pre-trend non-rejection cannot prove post-treatment parallel trends;
  • an RD bandwidth cannot prove no manipulation;
  • an IV first stage cannot prove exclusion;
  • synthetic-control pre-fit cannot prove the donor path remains valid.

Record all planned checks, their uncertainty, and what action each result triggers. Preserve unfavorable results and distinguish confirmatory from exploratory analyses.

An honest conclusion template​

For [target population], we estimated [estimand] for [defined treatment versus comparator] on [outcome and horizon] as [magnitude and uncertainty]. A causal interpretation requires [method-specific assumptions]. [Diagnostics and falsification results] reveal [supported or failed implications], while [unmeasured threats] remain untested. Sensitivity analysis shows [tipping point or range]. Therefore [decision implication]; the data do / do not identify the claimed effect.

“Not identified” is useful when overlap is absent, assignment cannot be made conditionally exchangeable, pre-trends or negative controls fail materially, manipulation or concurrent rules undermine RD, exclusion is implausible, or no defensible donor pool exists. You may still report descriptive patterns, forecast outcomes, estimate the effect of an encouragement, or propose new data and design.

Common mistakes​

  • Running many specifications and calling the range sensitivity analysis without a bias model.
  • interpreting robustness to analyst choices as robustness to hidden confounding.
  • converting a benchmark into proof that an omitted variable is implausible.
  • burying failed falsification tests in an appendix.
  • weakening the causal claim only after a reviewer discovers a violation.

Practice​

Choose one prior method and complete the conclusion template twice: once when diagnostics are consistent with the design and once when a decisive falsification check fails. The second version must recommend a concrete redesign or noncausal target.

Next steps​

Return to Causal Questions from Domain Knowledge for a new decision, or continue with From Notebook to Maintainable Project. The causal area is complete when method choice follows assignment logic rather than estimator fashion.

Primary sources​

Checked 2026-10-15: Rosenbaum, sensitivity analysis in observational studies, Cinelli and Hazlett, omitted-variable sensitivity, and Roth et al., sensitivity and modern DiD practice.