Units, Treatment, and Outcomes
“One row per student” does not settle the design. An experiment can have different units for eligibility, assignment, exposure, measurement, and analysis. Confusing them changes the estimand and usually understates uncertainty.
Prerequisites
Complete From Decision to Testable Hypothesis and keep its Study Sprint protocol sentence visible.
Learning goals
- Distinguish experimental units, assignment units, observation units, and the unit of analysis.
- Define treatment versions, comparator, uptake, outcomes, and measurement timing.
- Identify consent, exposure, interference, and measurement risks before randomization.
Map the units
For individual assignment, the eligible student is the experimental unit and assignment unit. The final-exam record is an observation about that student, and a student-level analysis can match the assignment. If whole tutorial sections are assigned, the section is the assignment unit while students remain observation units. Treating those students as independently randomized would create false precision.
Record at least:
| Design element | Study Sprint definition |
|---|---|
| Eligibility unit | Student registered before the assignment snapshot |
| Assignment unit | Student, unless spillovers require tutorial-section assignment |
| Exposure | Offer delivered, materials available, reminders sent |
| Observation unit | Student exam and burden records |
| Unit of analysis | Student with design-consistent standard errors, or cluster-level/cluster-aware analysis |
| Analysis population | All assigned eligible students for the intention-to-treat estimand |
The assignment unit controls the number of independent randomization units. More measurements within a student or cluster can improve outcome measurement, but they do not create more independent assignments.
Make treatment and comparator reproducible
“Study support” is not one treatment. Specify duration, content, delivery channel, availability, facilitator contact, reminders, and allowed co-interventions. Define usual resources just as carefully. Version drift during the experiment can violate consistency because the same treatment label then represents materially different experiences.
Record both assignment and exposure. Assignment supports the intention-to-treat effect; uptake can explain implementation but conditioning the primary comparison on uptake breaks randomization. A later per-protocol or complier analysis needs additional assumptions and is not a replacement for the assignment effect.
Outcomes, guardrails, and timing
Choose one primary outcome that answers the decision: the scheduled final-exam score at six weeks, measured identically in both arms. Define the scale, data source, valid range, time window, aggregation, and handling of missing exams. A convenient proxy such as clicks is not interchangeable with learning.
Secondary outcomes can explain mechanisms or durability. Guardrail metrics protect dimensions that should not worsen: withdrawal, reported burden, unequal access, privacy incidents, or severe subgroup deterioration. State thresholds and responses before launch. Stopping for harm can override the evidential stopping rule; operational teams should not wait for statistical significance when a credible safety threshold is crossed.
Ethics and interference
Informed consent should describe assignment, data use, foreseeable burdens, alternatives, withdrawal, and whom to contact, in language participants can understand. Consent alone does not make an avoidable exposure ethical. Minimize data and exposure, distribute burdens fairly, and use the appropriate local review process.
Peers can share materials, tutors may change behavior for everyone, and one student's participation can affect another's score. Document these interference paths. Possible responses include assigning intact groups, measuring network exposure, separating delivery, changing the estimand to a policy effect, or declining to run the experiment.
Common mistakes
- Equating a database row with an independent assignment unit.
- Describing only treatment and leaving the comparator variable.
- Using post-assignment engagement to define the primary analysis population.
- Choosing an easy-to-log proxy as the primary outcome without validating its decision relevance.
- Treating consent as permission to ignore spillovers, privacy, or unequal burden.
Practice
Draw five boxes for eligibility, assignment, exposure, observation, and analysis. Put the unit and timestamp in each. Add one interference arrow and one missing-outcome path, then say whether they change the design, estimand, or analysis.
Next steps
Randomization and Allocation defines the assignment mechanism and its audit trail.
Primary sources
Checked 2026-10-15: CONSORT 2010 statement, ICH E9(R1), estimands and intercurrent events, and The Belmont Report.