ORIGINAL COMPANY-CONTEXT PRACTICE
Analyze feedback from a self-selected survey
Original practice scenario: A synthetic consumer survey is sent after product registration, but only a small fraction respond. How should an analyst describe satisfaction results?
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
The responses describe respondents, not automatically the entire customer population. Registration and response behavior can both select a subgroup whose experiences differ from non-respondents.
Work through it
- Define the target population and sampling stages, then compare observable characteristics of invitees, respondents and the intended customer base.
- Report response rates and uncertainty by segment; use weighting only with defensible inclusion or response assumptions and adequate overlap.
- Run sensitivity analyses for plausible non-response patterns and distinguish descriptive survey associations from causal effects of product changes.
Tradeoffs to discuss
- Weighting can reduce measured imbalance but increases variance and cannot remove bias from unobserved factors without additional assumptions.
Common pitfalls
- Do not treat a narrow confidence interval as proof of representativeness.
- Avoid extrapolating beyond the population actually covered by the survey design.