Product Decisions and Measurement Questions
Product analytics should begin with a decision, not with whatever fields happen to be available. A measurement question states what must be observed, for whom, over what period, and how the answer could change an action. The resulting metric is evidence for that question; it is not the decision itself.
Prerequisites
Be comfortable with tabular data, timestamps, proportions, and the distinction between association and causation. The Experimentation & Causal Inference path is the prerequisite for intervention claims.
Learning goals
- Translate a product decision into a population, behavior, window, and comparison.
- Separate collection, measurement, prediction, and causal inference.
- Name the unit of analysis before choosing an aggregation.
- Reject measurements that collect more personal data than the question needs.
From decision to evidence
Suppose a team must decide whether to repair the first signup step. “Increase growth” is not yet measurable. A better chain is:
| Layer | Declared statement |
|---|---|
| Decision | Keep the flow, repair the first step, or investigate another bottleneck. |
| Measurement question | Among eligible actors who start signup on 1 October UTC, what share has a valid account_created event by the reporting cutoff? |
| Unit of analysis | Resolved actor and signup attempt. |
| Signal | signup_started followed by the server-confirmed account event. |
| Metric | verified_activation_rate version 1.0.0. |
| Action boundary | A low rate identifies where to investigate; it does not prove that a proposed redesign will improve activation. |
This wording exposes denominator, outcome, time, and an action. It also makes missing instrumentation visible. Recording an account outcome on the client would answer a different question from recording a committed account on the server.
Collection asks which events enter the log. Identity resolution decides which records may represent the same actor. Measurement applies a declared aggregation. Inference asks what would happen under a different action. The fixture in this path supports the first three; it contains no randomized assignment or identification strategy, so every result remains descriptive.
Privacy belongs in the question. The worked metric needs event time, an approved actor key, event name, schema validity, receipt time, and consent state. It does not need message text, precise location, contact details, or a permanent cross-context identity. Purpose limitation and data minimization are design constraints, not cleanup tasks.
Worked example
question = {
"decision": "improve_onboarding",
"text": "Do eligible starters create an account?",
"metric": "verified_activation_rate",
"design": "descriptive",
}
print(f"decision={question['decision']} question={question['text']}")
print(f"descriptive_metric={question['metric']} causal_claim={question['design'] == 'causal'}")
Expected output:
decision=improve_onboarding question=Do eligible starters create an account?
descriptive_metric=verified_activation_rate causal_claim=False
The boolean is deliberately false. A dashboard movement can motivate a hypothesis, but estimating the effect of a redesign requires an experiment or a defensible observational identification design.
Limitations and common mistakes
- “Users are less engaged” lacks a population, behavior, window, and comparison.
- Choosing a metric because it is easy to query reverses the decision-to-evidence order.
- Treating a changed metric as the effect of a release ignores seasonality, selection, concurrent changes, and measurement drift.
- Collecting rich attributes “for later” violates data minimization and makes deletion, access control, and incident response harder.
- A precise value can still answer the wrong question because the unit of analysis or event trigger is wrong.
Practice
Write a decision and measurement question for improving report export. Declare the actor population, observation window, successful behavior, reporting cutoff, and what evidence would trigger investigation. List two tempting properties that the question does not justify collecting.
Next steps
Define the records and grains behind the question in users, sessions, events, and business entities.
Primary sources
Checked 2026-10-19: Rodden, Hutchinson, and Fu, Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications; W3C, Privacy Principles; Hernán and Robins, Causal Inference: What If.