Power, Minimum Effect, and Sample Size

Power planning asks how often a design would meet a stated evidence rule under explicit data-generating assumptions. It is not a promise about one experiment and it cannot turn an unimportant effect into an important one.

Prerequisites

Complete Blocking, Stratification, and Pairing. Fix the primary estimand, outcome scale, analysis, and assignment structure first.

Learning goals

• Distinguish a smallest worthwhile effect from a minimum detectable effect.
• Calculate and challenge sample-size assumptions.
• Plan sensitivity analyses for variance, attrition, clustering, noncompliance, and multiplicity.

Start with the decision threshold

The smallest worthwhile effect comes from consequences: Study Sprint must improve the score enough to justify staff time and student burden. The minimum detectable effect (MDE) is a property of a proposed design at chosen Type I error and power. It is a decision input, not a value selected solely to justify an affordable sample.

If only a 7-point effect is detectable but a 4-point effect would change the decision, label the design underpowered for the decision. Options include improving outcome precision, using strong baseline adjustment, extending recruitment, simplifying the question, treating the study as a pilot for process uncertainty, or not running it. Renaming 7 points “meaningful” hides the constraint.

A transparent approximation

For equal-sized independent arms, a continuous outcome, common standard deviation sd, two-sided alpha, and target power, a normal approximation is

$$n_{\text{per arm}} = 2\left(\frac{(z_{1-\alpha/2}+z_{\text{power}})\sigma}{\Delta}\right)^2.$$

import math
from statistics import NormalDist

effect = 4.0
sd = 10.0
alpha = 0.05
power = 0.80
normal = NormalDist()
z_alpha = normal.inv_cdf(1 - alpha / 2)
z_power = normal.inv_cdf(power)
per_arm = math.ceil(2 * ((z_alpha + z_power) * sd / effect) ** 2)

print(f"assumed effect={effect:.1f} sd={sd:.1f} alpha={alpha:.2f} power={power:.2f}")
print(f"required per arm={per_arm} total={2 * per_arm}")

This is a planning approximation, not a universal formula. Binary, count, ratio, heavy-tailed, clustered, repeated-measure, unequal-allocation, and sequential designs require design-consistent calculations. Simulation is often clearest when the analysis is complex.

Build an assumption table

Record the effect and why it matters, outcome variance and its source, alpha and sidedness, target power, allocation ratio, analysis model, baseline correlation, cluster sizes and intracluster correlation if relevant, expected missingness, noncompliance, multiplicity, and rounding. Report a grid rather than one brittle number.

Inflating the result by 1 / (1 - attrition) assumes the retained observations still support the estimand. It does not repair informative missingness. Likewise, adding participants does not fix biased measurement, interference, broken allocation, or an irrelevant outcome.

Power is effect-specific: 80% power at 4 points differs from the probability of detecting 2 points. Post-hoc “observed power” computed from the observed effect adds no useful evidence beyond the estimate and interval.

Common mistakes

• Solving for the effect after fixing the available sample, then calling it meaningful.
• Using a standard deviation from a different scale or selected subgroup without sensitivity analysis.
• Counting observations rather than independent assignment units.
• Ignoring attrition, clustering, multiplicity, or the planned covariate adjustment.
• Treating 80% power as an 80% probability that the current hypothesis is true.

Practice

Create a table crossing effects of 2, 4, and 6 points with standard deviations of 8, 10, and 12. Mark the worthwhile effect independently, identify feasible sample sizes, and write the design decision if none provides adequate power.

Next steps

Cluster-Randomized Experiments shows why the number and size of assignment groups—not merely the student count—control information.

Primary sources

Checked 2026-10-15: Neyman and Pearson, “On the Problem of the Most Efficient Tests of Statistical Hypotheses”, NIST/SEMATECH, selecting sample sizes, and CONSORT 2010 statement.