Skip to main content

Peeking, Stopping, and Sequential Designs

Looking at accumulating results and stopping when an ordinary p-value first crosses 0.05 changes the experiment. The final evidence rule is no longer a single fixed-sample test, so its false-positive probability is no longer 5%.

Prerequisites​

Complete Factorial Experiments. Be comfortable with Type I error, power, and a fixed-sample stopping rule.

Learning goals​

  • Explain why optional stopping with repeated fixed-sample tests inflates error.
  • Distinguish a fixed design, naive peeking, and a valid sequential design.
  • Prespecify looks, boundaries, decisions, estimators, guardrails, and operational roles.

A local null simulation​

This simulation generates five independent standard-normal increments under no effect. Each look uses all increments accumulated so far. The fixed design tests only at look five; naive peeking uses 0.05 at every look; the conservative sequential rule divides alpha across five looks with Bonferroni.

import random
from statistics import NormalDist

simulations = 20_000
looks = 5
rng = random.Random(3680)
normal = NormalDist()
fixed_boundary = normal.inv_cdf(0.975)
sequential_boundary = normal.inv_cdf(1 - 0.05 / (2 * looks))
fixed = naive = sequential = 0

for _ in range(simulations):
    total = 0.0
    crossed_naive = crossed_sequential = False
    for look in range(1, looks + 1):
        total += rng.normalvariate(0, 1)
        z_value = total / (look ** 0.5)
        crossed_naive |= abs(z_value) >= fixed_boundary
        crossed_sequential |= abs(z_value) >= sequential_boundary
    fixed += abs(z_value) >= fixed_boundary
    naive += crossed_naive
    sequential += crossed_sequential

print(f"null simulations={simulations} looks={looks}")
print(f"fixed false-positive rate={fixed / simulations:.3f}")
print(f"naive peeking rate={naive / simulations:.3f}")
print(f"bonferroni sequential rate={sequential / simulations:.3f}")

Monte Carlo rates vary with the seed and number of simulations; the verifier fixes both. Bonferroni is easy to explain but conservative because looks are correlated. Group-sequential boundaries such as Pocock or O'Brien–Fleming, alpha-spending functions, always-valid methods, and Bayesian decision rules offer different trade-offs. Their validity depends on the exact prospective design and assumptions, not the word “sequential.”

What to prespecify​

Before data accrue, specify the maximum information or sample size, number and timing of looks, statistic, efficacy boundary, futility rule, estimator and interval adjusted for the stopping process, missing-data handling, and who may see interim results. Simulate Type I error, power, expected sample size, stopping probabilities, and behavior under delayed outcomes or overrun.

A valid sequential design controls its chosen error criterion, such as the family-wise Type I error, under stated conditions. Changing the next look because the current result is “almost significant,” adding unplanned arms, or continuing after a binding rule can invalidate those operating characteristics.

Harm, futility, and benefit are different decisions​

Stopping for harm is an ethical guardrail and may require rapid action based on multiple sources, not merely the efficacy p-value. A credible harm threshold should trigger the prespecified response and oversight path. Futility can reduce expected exposure when the planned evidence target is unlikely to be reached, but “not promising” is not proof of no effect. Early efficacy stopping may leave limited information about harms, durability, and secondary outcomes and can exaggerate conventional effect estimates.

Operational dashboards may monitor enrollment, delivery, data quality, privacy incidents, and guardrails without exposing comparative efficacy to the implementation team. Separate access and roles so ordinary operations do not become undisclosed outcome peeking.

Common mistakes​

  • Checking an ordinary 0.05 test daily and reporting only the crossing day.
  • Calling an unplanned early stop a sequential design.
  • Prespecifying boundaries but using an ordinary fixed-sample interval after stopping.
  • Combining efficacy, futility, and harm into one vague “stop if results are clear” rule.
  • Ending follow-up at enrollment stop when delayed outcomes and safety monitoring still matter.

Practice​

Write a two-look Study Sprint design. Name the information times, efficacy and futility boundaries, harm guardrail, maximum sample, interim access role, adjusted final analysis, delayed-outcome plan, and simulation scenarios required before launch.

Next steps​

Preregistration and Analysis Plans assembles the decisions into a dated, auditable protocol while leaving room for transparently labeled exploration.

Primary sources​

Checked 2026-10-15: Pocock, “Group Sequential Methods in the Design and Analysis of Clinical Trials”, FDA, Adaptive Design Clinical Trials guidance, and O'Brien and Fleming, “A Multiple Testing Procedure for Clinical Trials”.