ORIGINAL TECHNICAL PRACTICE
Choose a realistic validation split
You have repeated observations for customers collected over18months. How would you choose training and test splits?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
The split should represent the prediction task. Whether future observations of existing customers or entirely unseen customers matter determines the time and entity boundaries the evaluation must enforce.
Work through it
- Write down the prediction time, the available history and the intended deployment population. Decide whether customer overlap between train and test is legitimate.
- Use chronological evaluation for future prediction and grouped splits for unseen-customer generalization. Combine these constraints when both are required.
- Fit preprocessing only on each training partition. Ensure labels and aggregates use only information available before the prediction moment, then preserve a final untouched test period.
Tradeoffs to discuss
- Strict grouping and time separation reduce the amount of usable training data. They often produce a lower but more realistic estimate than random row splits.
Common pitfalls
- Randomly splitting rows can put nearly identical customer records on both sides. Global feature aggregation can leak future behavior even after the rows are split.