ORIGINAL TECHNICAL PRACTICE
Detect leakage in a model pipeline
Your classifier performs extremely well offline and poorly after release. How would you investigate data leakage?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Check whether any training input, preprocessing step or evaluation split uses information that would be unavailable at the decision time. Leakage can arise even when the target column itself was removed.
Work through it
- Draw the event timeline and identify the exact prediction moment. Audit features such as later status changes, refunds or labels added after that moment.
- Check entity overlap and near-duplicate records across splits. A random row split is insufficient when records from the same customer or device must remain grouped.
- Fit imputers, scalers, feature selectors and encoders on training data within each fold. Compare a leakage-resistant time or group split with the original result.
Tradeoffs to discuss
- A stricter split often reduces the score while better matching the deployment task. Less training data may be worth a more credible evaluation.
Common pitfalls
- Fixing only an obvious target proxy can leave entity and preprocessing leakage. A pipeline helps prevent some leakage but cannot repair a split that violates time or entity boundaries.