Factorial Experiments

A factorial experiment varies two or more factors together. A 2×2 Study Sprint design can randomize the core program on or off and reminders on or off, producing four cells. The design can estimate each factor's average effect and whether their combination behaves differently from the sum of separate effects.

Prerequisites

Complete Cluster-Randomized Experiments. Return to individual assignment for this lesson unless the protocol explicitly combines factorial and cluster assignment.

Learning goals

• Define factors, levels, cells, main effects, and interactions.
• Calculate effects using all relevant randomized cells.
• Plan allocation, power, interpretation, and guardrails for combinations.

The 2×2 design

Core Study Sprint	Reminders	Cell meaning
Off	Off	Usual resources
On	Off	Core package only
Off	On	Reminders only
On	On	Combined package

Under balanced randomization, the core-program main effect averages its contrast across both reminder levels. The reminder main effect does the symmetric comparison. On an additive outcome scale, the interaction is a difference in differences.

cell_mean = {
    (0, 0): 70.0,
    (1, 0): 72.5,
    (0, 1): 69.5,
    (1, 1): 75.0,
}

sprint = ((cell_mean[1, 0] + cell_mean[1, 1]) - (cell_mean[0, 0] + cell_mean[0, 1])) / 2
reminder = ((cell_mean[0, 1] + cell_mean[1, 1]) - (cell_mean[0, 0] + cell_mean[1, 0])) / 2
interaction = cell_mean[1, 1] - cell_mean[1, 0] - cell_mean[0, 1] + cell_mean[0, 0]

print(f"study-sprint main effect={sprint:.1f}")
print(f"reminder main effect={reminder:.1f}")
print(f"interaction={interaction:.1f}")

The positive interaction says the reminder works differently when the core package is present on this scale. It does not reveal a mechanism by itself. Interactions depend on the outcome scale: an additive interaction can differ from an interaction on a ratio or log scale.

Assignment and analysis implications

Randomize participants across all four cells, often equally, and store both factor assignments. Analyze with a model containing each factor and their interaction term, or use equivalent planned cell contrasts. Dropping the interaction without justification can misrepresent both main effects when the combination is important.

Power the design for the contrast that drives the decision. Detecting an interaction usually requires more information than detecting a main effect of the same apparent size. If the decision is specifically whether reminders add value to Study Sprint, power that simple effect or interaction rather than celebrating an adequately powered average reminder effect.

Factorial efficiency relies on exposing each unit to a valid combination. Do not cross factors when one combination is unsafe, impossible, or meaningless. Prespecify combination-specific guardrails because two low-burden components may interact to create an unacceptable burden.

More factors and fractional designs

A full $2^k$ design grows quickly. Fractional factorial designs run selected combinations and deliberately alias some effects. They can be useful for screening when higher-order interactions are plausibly negligible, but aliasing is a design assumption, not free information. Document which effects cannot be separated.

Common mistakes

• Comparing only the “both on” and “both off” cells and calling the result two component effects.
• Interpreting a main effect without stating the averaging distribution over the other factor.
• Omitting the interaction term because its p-value is inconvenient.
• Powering only main effects when the product decision depends on synergy or antagonism.
• Running combinations whose joint burden or exposure was never ethically reviewed.

Practice

Draft a 2×2 table for Study Sprint and optional tutoring. State the three planned contrasts, the decision attached to each, allocation probabilities, primary outcome, interaction scale, combination guardrails, and which contrast determines sample size.

Next steps

Peeking, Stopping, and Sequential Designs explains why repeated looks are another design choice rather than an informal monitoring habit.

Primary sources

Checked 2026-10-15: NIST/SEMATECH, full factorial designs, NIST/SEMATECH, blocking full factorial designs, and Fisher, The Design of Experiments.