Skip to main content

Binary, Count, and Ratio Metrics

The assignment unit stays the student even when the outcome is a conversion indicator, a count of sessions, or a ratio of questions to sessions. Metric construction changes the scientific question; adding event rows does not add independent randomized units.

Before analysis, freeze each metric's numerator, denominator, time window, zero handling, eligibility, and aggregation. The fixture supports three distinct ITT estimands: the risk difference in activation, the difference in mean sessions per randomized student, and the difference in questions per session across all activity generated by students in each assigned arm.

Prerequisites​

Complete Regression Adjustment for Experiments and retain user_id as the assignment key.

Learning goals​

  • Choose effect scales that answer a stated decision.
  • Keep student-level dependence in standard errors for binary, count, and ratio metrics.
  • Distinguish a ratio of totals from a mean of student-level ratios.

Three metrics, three estimands​

import csv
import math
import statistics

with open("fixtures/experimentation-causal-inference/ab-test-users.csv", newline="") as handle:
    rows = list(csv.DictReader(handle))
groups = {arm: [row for row in rows if row["assigned_arm"] == arm]
          for arm in ("control", "treatment")}

def mean_effect(field):
    values = {arm: [float(row[field]) for row in group] for arm, group in groups.items()}
    effect = statistics.mean(values["treatment"]) - statistics.mean(values["control"])
    se = math.sqrt(sum(statistics.variance(v) / len(v) for v in values.values()))
    return effect, effect - 1.96 * se, effect + 1.96 * se

activation = mean_effect("activated")
sessions = mean_effect("sessions")
ratios, variances = {}, {}
for arm, group in groups.items():
    numerator = [float(row["questions_answered"]) for row in group]
    denominator = [float(row["sessions"]) for row in group]
    ratios[arm] = sum(numerator) / sum(denominator)
    residual = [n - ratios[arm] * d for n, d in zip(numerator, denominator, strict=True)]
    variances[arm] = statistics.variance(residual) / (
        len(group) * statistics.mean(denominator) ** 2
    )
ratio_effect = ratios["treatment"] - ratios["control"]
ratio_se = math.sqrt(sum(variances.values()))

print(f"activation risk difference={activation[0]:.3f} 95% CI=({activation[1]:.3f}, {activation[2]:.3f})")
print(f"sessions per student difference={sessions[0]:.2f} 95% CI=({sessions[1]:.2f}, {sessions[2]:.2f})")
print(f"questions per session difference={ratio_effect:.3f} 95% CI=({ratio_effect - 1.96 * ratio_se:.3f}, {ratio_effect + 1.96 * ratio_se:.3f})")

The ratio calculation uses student-level numerator and denominator contributions in a delta-method approximation. Expanding the aggregates into session rows and applying an independent-row test would understate uncertainty because the assignment and behavioral dependence live at student level. A student-cluster bootstrap or assignment-aware randomization procedure is another defensible route.

For binary outcomes, risk difference is directly actionable; risk ratio and odds ratio answer different questions and need their own intervals. Counts often have skew, zeros, and overdispersion, so a difference in means can be robust as an estimand even when a Poisson likelihood is implausible. For ratios, sum(questions) / sum(sessions) weights high-activity students more than mean(questions / sessions); neither is automatically correct.

Denominator changes are effects too​

A treatment can change both questions and sessions. A higher questions-per-session ratio could reflect more questions, fewer sessions, or both. Always report numerator and denominator components beside the ratio. Define what happens when a student has zero sessions before assignment; dropping those students creates a post-treatment subset.

Common mistakes​

  • Treating events as if they were randomized independently.
  • Calling an odds ratio a percentage-point change.
  • Choosing ratio-of-totals versus mean-of-ratios after seeing which looks better.
  • Dropping zero-denominator students without defining the target population.
  • Applying a default Poisson model despite excess zeros or overdispersion.

Practice​

For each fixture metric, write one sentence naming the population weighting and effect scale. Then explain how a treatment-caused denominator change would alter interpretation.

Next steps​

Multiple Metrics and Multiple Testing organizes a metric family before interpreting several results.

Primary sources​

Checked 2026-10-15: Deng et al., “Applying the Delta Method in Metric Analytics”, Deng et al., “Consistent Transformation of Ratio Metrics”, and CONSORT 2025 explanation and elaboration.