ORIGINAL COMPANY-CONTEXT PRACTICE
Evaluate a simpler onboarding flow
Original practice scenario inspired by Commonwealth Bank of Australia's public business context; this is not a reported interview question or a description of its internal systems. A fictional banking app tests a shorter onboarding flow. Completion improves, but support contacts and later document corrections also rise. How would you decide whether the experiment supports rollout?
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
Predefine the primary outcome and guardrails across the whole onboarding journey. Faster completion is useful only in the context of eligibility, downstream correction, and customer-support costs.
Work through it
- Randomize at the applicant level and keep assignment stable across devices. Define the eligible population and measure intent-to-treat completion within a fixed follow-up window.
- Check assignment balance, event completeness, and repeat applicants. Report absolute effects and uncertainty for completion, corrections, and support contacts.
- Inspect whether follow-up is mature enough to observe delayed issues. Decide using prespecified guardrails and a reversible staged rollout rather than choosing only favorable metrics.
Tradeoffs to discuss
- A shorter test delivers answers sooner but may miss delayed operational consequences. Multiple guardrails need a clear decision policy, not selective significance hunting.
Common pitfalls
- Analyzing only completed applicants conditions on an outcome changed by treatment.
- A relative lift without base rates hides practical effect size.