ORIGINAL TECHNICAL PRACTICE
Backtest a forecast without looking ahead
You must forecast weekly demand four weeks ahead. How would you compare candidate models?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Use evaluation origins that preserve time order and match the four-week forecast horizon. At each origin, train using only data available then and score predictions against later observations.
Work through it
- Establish a seasonal or simple carry-forward baseline. Define whether the model predicts each of the next four weeks or their combined total.
- Use rolling or expanding training windows across multiple origins. Recreate feature availability, reporting delays and any known future covariates for each origin.
- Compare errors by horizon, season, product segment and demand volume. Assess bias and operational consequences alongside an aggregate metric.
Tradeoffs to discuss
- Expanding windows use more history; rolling windows can adapt to changing patterns. The best choice depends on how stable the demand-generating process is.
Common pitfalls
- Random cross-validation leaks temporal information. A percentage error metric can behave badly at zero demand, and tuning repeatedly on one holdout period can overfit it.