Skip to main content

Multiple Metrics and Multiple Testing

An experiment dashboard can calculate hundreds of contrasts. That does not create hundreds of equally confirmatory answers. Before assignment, define one primary estimand, a small family of key secondary estimands, guardrails, and clearly exploratory diagnostics.

For Study Sprint, the primary estimand is the ITT mean score effect. Activation and sessions are secondary behavioral outcomes. The adverse-event indicator is a guardrail: it can block adoption even when it is not part of a symmetric “benefit family.”

Prerequisites​

Complete Binary, Count, and Ratio Metrics and return to the frozen analysis plan.

Learning goals​

  • Define the hypothesis family from the decision, not from observed p-values.
  • Apply a transparent family-wise error procedure to confirmatory metrics.
  • Keep guardrail governance and exploratory evidence visible.

One declared family​

This example computes large-sample two-sided p-values for three prespecified benefit metrics, then applies Holm's step-down procedure. The fixed numerical output is an audit of the procedure, not a recommendation to base the decision on adjusted p-values alone.

import csv
import math
import statistics
from statistics import NormalDist

with open("fixtures/experimentation-causal-inference/ab-test-users.csv", newline="") as handle:
    rows = list(csv.DictReader(handle))
groups = {arm: [row for row in rows if row["assigned_arm"] == arm]
          for arm in ("control", "treatment")}

def estimate(field, observed=lambda row: True):
    values = {arm: [float(row[field]) for row in group if observed(row)]
              for arm, group in groups.items()}
    effect = statistics.mean(values["treatment"]) - statistics.mean(values["control"])
    se = math.sqrt(sum(statistics.variance(v) / len(v) for v in values.values()))
    return effect, se

tests = {
    "score": estimate("final_score", lambda row: bool(row["final_score"])),
    "activation": estimate("activated"),
    "sessions": estimate("sessions"),
}
p_values = {
    name: 2 * (1 - NormalDist().cdf(abs(effect / se)))
    for name, (effect, se) in tests.items()
}
ordered = sorted(p_values, key=p_values.get)
adjusted, running = {}, 0.0
for index, name in enumerate(ordered):
    candidate = min(1.0, (len(ordered) - index) * p_values[name])
    running = max(running, candidate)
    adjusted[name] = running

for name in ("score", "activation", "sessions"):
    print(f"{name}: raw={p_values[name]:.3f} Holm={adjusted[name]:.3f}")

Holm controls family-wise Type I error under arbitrary dependence and is uniformly at least as powerful as single-step Bonferroni. Other valid strategies include a gatekeeping hierarchy, a single primary outcome with explicitly descriptive secondary intervals, or false-discovery-rate control for a genuine discovery program. The choice belongs in the plan.

Multiplicity is not erased by calling metrics “supporting,” inspecting many slices, trying transformations, or showing only favorable dashboards. Report every planned metric, its effect size and interval, the family definition, the adjustment, and any deviations.

Guardrails are governance constraints​

A harm metric is not automatically interchangeable with a benefit metric. Predefine which harm triggers pause, investigation, or rejection, how delayed harm is handled, and who can act. A wide interval around a rare adverse event is not reassurance. Report the control and treatment risks and the uncertainty even when a formal threshold is not crossed.

Exploratory metrics can explain mechanisms and suggest later experiments. Label them exploratory, show how many analyses were searched, and avoid retroactively promoting a favorable result to the original primary question.

Common mistakes​

  • Correcting only the p-values shown in the final report.
  • Treating correlated metrics as if multiplicity disappears.
  • Declaring success when any one metric crosses 0.05.
  • Using a benefit correction to weaken a safety stop.
  • Hiding null or harmful planned outcomes.

Practice​

Draft a one-page metric map with the score estimand, secondary behavior family, guardrails, and exploratory diagnostics. For every branch, state its reporting and decision rule before seeing outcomes.

Next steps​

Missing Data and Attrition handles the students whose primary outcomes were not observed.

Primary sources​

Checked 2026-10-15: Holm, “A Simple Sequentially Rejective Multiple Test Procedure”, FDA, Multiple Endpoints in Clinical Trials, and CONSORT 2025 explanation and elaboration.