ORIGINAL TECHNICAL PRACTICE
Monitor a model after deployment
Which changes would you monitor after launching a model, and how would you distinguish data drift from worse task performance?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Input drift describes changes in feature or request distributions; concept drift concerns changes in the relationship to the target. Neither can be inferred solely from traffic volume, and performance measurement needs suitable outcomes.
Work through it
- Monitor schema validity, missingness, feature distributions, prediction distributions and operational metrics. Segment by meaningful groups to detect localized changes.
- Collect delayed labels or adjudicated examples where available and compare task metrics against a stable baseline. Account for label delay and changing sampling before interpreting the trend.
- Set investigation thresholds, a review cadence and rollback or fallback criteria. Re-evaluate retraining candidates on recent and historical scenarios to expose regressions.
Tradeoffs to discuss
- Fast proxy metrics can detect problems before labels arrive but are not a replacement for outcome-based evaluation. Excessive alerts can consume the team’s attention.
Common pitfalls
- A statistically detectable input change is not automatically harmful. Retraining on every drift signal can amplify feedback loops or introduce unreviewed behavior.