Original practice scenario inspired by Commonwealth Bank of Australia's public business context; this is not a reported interview question or a description of its internal systems. A fictional banking app tests a shorter onboarding flow. Completion improves, but support contacts and later document corrections also rise. How would you decide whether the experiment supports rollout?
Read model answer +
Predefine the primary outcome and guardrails across the whole onboarding journey. Faster completion is useful only in the context of eligibility, downstream correction, and customer-support costs.
Work through it
Randomize at the applicant level and keep assignment stable across devices. Define the eligible population and measure intent-to-treat completion within a fixed follow-up window.
Check assignment balance, event completeness, and repeat applicants. Report absolute effects and uncertainty for completion, corrections, and support contacts.
Inspect whether follow-up is mature enough to observe delayed issues. Decide using prespecified guardrails and a reversible staged rollout rather than choosing only favorable metrics.
Tradeoffs to discuss
A shorter test delivers answers sooner but may miss delayed operational consequences. Multiple guardrails need a clear decision policy, not selective significance hunting.
Common pitfalls
Analyzing only completed applicants conditions on an outcome changed by treatment.
A relative lift without base rates hides practical effect size.
Original practice scenario inspired by Commonwealth Bank of Australia's public business context; this is not a reported interview question or a description of its internal systems. A synthetic account ledger has booking_time, effective_day, amount_cents, and correction relationships. Explain how to produce a reproducible statement for a historical cutoff without accidentally including later-known corrections.
Read model answer +
Specify whether the statement represents knowledge at publication or a later restatement. Booking time controls what was known; effective day controls the business period to which an entry belongs.
Work through it
Filter entries to booking_time at or before the recorded statement cutoff, then apply the documented effective-day range. Keep both timestamps in the audit trail.
Aggregate signed ledger entries, treating reversals and corrections according to their explicit relationships. Do not silently overwrite original posted amounts.
Persist statement parameters and source watermark. Test a late correction effective in the earlier month, an exact cutoff boundary, and a reversal arriving after publication.
Tradeoffs to discuss
Immutable statement snapshots improve reproducibility but require a separate restatement process when later information changes the period.
Common pitfalls
Filtering only by effective day leaks later knowledge into the original statement.
Summing both an overwritten value and its adjustment can double-count the correction.
