ORIGINAL TECHNICAL PRACTICE
Design an A/B test before looking at results
A product team wants to test a redesigned onboarding flow. What decisions should be made before the experiment starts?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Define the hypothesis, primary outcome, unit of randomization and analysis plan before inspecting results. The design should connect the product change to a measurable effect while protecting against avoidable bias.
Work through it
- Choose an outcome with a clear denominator and observation window, such as completion within seven days of assignment. Specify eligibility and treatment exposure.
- Randomize at a unit that limits interference, often a user or account. Estimate sample size from a meaningful effect size and expected variability, and choose a stopping rule.
- Track assignment balance, sample-ratio mismatch, missing outcomes and guardrail metrics. Analyze by assignment when that is the intended estimand, and report effect size with uncertainty.
Tradeoffs to discuss
- A longer observation window may capture more meaningful outcomes while delaying a decision. Multiple variants and metrics require an explicit multiplicity strategy.
Common pitfalls
- Repeated unplanned significance checks can inflate false positives. Excluding users because of behavior caused by treatment can introduce selection bias.