ORIGINAL TECHNICAL PRACTICE
Investigate training-serving population mismatch
A model trained on customers who completed an application performs poorly on all applicants. What would you investigate?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
The training data may represent a selected population rather than the deployment population. Completion can depend on characteristics related to the outcome, leaving important cases missing or underrepresented.
Work through it
- Compare eligibility, feature availability and outcome observability between completers and non-completers. Identify where the populations differ and where the training set has little support.
- Define the target outcome for all applicants and determine whether it can be validly observed. Explore improved data collection before choosing a statistical correction.
- Evaluate on a representative sample where possible. If applying weighting or another adjustment, state its assumptions and inspect extreme weights and unsupported regions.
Tradeoffs to discuss
- Weighting can help under suitable assumptions but cannot invent outcomes for completely unobserved regions. Broader data collection may be slower while addressing the root problem.
Common pitfalls
- High accuracy on another sample of completers does not establish deployment validity. Filling unobserved outcomes with a convenient label can introduce systematic error.