ORIGINAL COMPANY-CONTEXT PRACTICE
Evaluate a synthetic transaction-alert model with delayed labels
Original practice scenario: A transaction simulator flags suspicious activity, but confirmed outcomes arrive weeks later and investigators review mostly high-scoring cases.
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
Delayed and selective labels distort apparent precision and recall. A credible evaluation must describe which outcomes are observable and avoid treating unreviewed transactions as verified negatives.
Work through it
- Freeze features at decision time and record model scores, review selection and eventual outcome timestamps in a versioned evaluation dataset.
- Use sufficiently matured cohorts, assess reviewed and representative audit samples separately, and report uncertainty and review-capacity constraints.
- Monitor drift in score distributions and segment outcomes, testing threshold policies against operational cost and appeal or correction pathways.
Tradeoffs to discuss
- Random audit sampling improves coverage but consumes review resources; importance weighting depends on known nonzero selection probabilities.
Common pitfalls
- Do not leak investigator conclusions into training features.
- Avoid claiming unbiased recall when the unreviewed population's true outcomes are unknown.