ORIGINAL TECHNICAL PRACTICE
Handle missing values without leaking information
A feature is missing for40% of records. How would you decide whether and how to use it?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Investigate why the feature is missing and whether the same pattern will occur at prediction time. Missingness may encode collection processes, but it can also reflect unavailable future information or a changing pipeline.
Work through it
- Compare missingness across time, sources, labels and relevant groups. Verify whether the absence comes from a legitimate state, a system failure or a business rule.
- Choose an imputation or model strategy suited to the variable and task. Fit imputation parameters on training data only and consider a missingness indicator when justified.
- Evaluate performance and calibration separately for missing and observed cases. Test how the model behaves if collection coverage changes after deployment.
Tradeoffs to discuss
- Dropping incomplete rows can simplify analysis while altering the population. Imputation preserves observations but introduces assumptions that should be assessed.
Common pitfalls
- A single global mean fitted before splitting leaks information. Treating missing values as zero can create a meaning that the source never intended.