Skip to main content

From Decision to Testable Hypothesis

An experiment is a decision instrument, not a machine for producing small p-values. Begin with the action someone may take, the people affected, and the effect that would change that action. Only then define a hypothesis and statistical procedure.

We continue the Study Sprint scenario: a college can offer a six-week study-support package before the final exam. The decision is whether to offer it next term, not whether last term's participants happened to score higher.

Prerequisites​

Complete Causal Questions from Domain Knowledge. You should already be able to state a population, intervention, comparator, outcome, horizon, estimand, and identification assumptions.

Learning goals​

  • Convert a practical decision into a testable causal claim.
  • Separate the estimand, decision threshold, hypothesis, and evidence rule.
  • Draft the ethical and operational constraints before exposure begins.

A design chain that starts with the decision​

Write the protocol in this order:

  1. Decision: offer, revise, or decline Study Sprint next term.
  2. Stakeholders and costs: eligible students experience time, attention, privacy, and possible learning effects; staff provide the program.
  3. Target population: students eligible at registration time next term.
  4. Intervention and comparator: assignment to a defined offer versus usual resources.
  5. Estimand: the intention-to-treat ATE of the offer on the scheduled final-exam score at six weeks.
  6. Decision threshold: a gain of 4 points is the smallest benefit that would justify the cost and burden, subject to guardrails.
  7. Evidence rule: estimate the effect and uncertainty using the prespecified analysis; do not replace the decision threshold with “p-value below 0.05.”

The testable hypothesis is about the estimand. A two-sided null might be that the assignment effect is zero. A decision rule is richer: adopt only if the estimate is compatible with a worthwhile benefit and attendance burden, withdrawals, or subgroup harms do not cross prespecified guardrails. Failure to reject zero is not proof of no useful effect; statistical significance is not proof that the effect is useful.

Design before testing​

The protocol must name the assignment unit, treatment versions, allocation ratio and mechanism, primary outcome and measurement time, analysis population, expected variance, minimum detectable effect, sample-size assumptions, missing-data strategy, guardrail metrics, and stopping rule. These choices determine what the test means.

If the intended intervention could spread between peers, define the interference model before choosing individual randomization. If access cannot ethically be withheld, redesign the comparison rather than hiding the constraint. If informed consent, privacy, or oversight requirements are uncertain, obtain appropriate local review; this lesson does not provide domain-specific medical or legal approval guidance.

A protocol sentence​

Among students eligible before assignment, compare assignment to a defined six-week Study Sprint offer with assignment to usual resources, estimate the six-week intention-to-treat mean score effect, and decide whether a benefit of at least 4 points is plausible without violating prespecified burden and harm guardrails.

Every noun in that sentence becomes a field in the plan. “Students who used the program” would silently change the estimand by conditioning on post-assignment uptake.

Common mistakes​

  • Starting with a preferred test and inventing a causal question around it.
  • Treating the null value, smallest worthwhile effect, and minimum detectable effect as the same quantity.
  • Writing an intervention label without specifying the experience it represents.
  • Selecting a primary outcome after inspecting which metric moved.
  • Treating ethics as a paragraph added after power and allocation are fixed.

Practice​

Choose a reversible decision. Write one sentence containing the population, assignment, comparator, outcome, horizon, estimand, worthwhile effect, and guardrails. Then list one ethical or operational constraint that could change the assignment design.

Next steps​

Units, Treatment, and Outcomes turns each noun in the protocol into something assignable and measurable.

Primary sources​

Checked 2026-10-15: ICH E9(R1), estimands and sensitivity analysis, CONSORT 2010 statement, and The Belmont Report.