ORIGINAL COMPANY-CONTEXT PRACTICE
Evaluate a diagnostic-model score under population change
Original practice scenario: Using synthetic data for a hypothetical diagnostic research classifier, explain how a different outcome prevalence can change precision even if sensitivity and specificity remain similar.
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
Precision depends on how common the target outcome is in the evaluated population, as well as the classifier's conditional error rates. One reported metric is therefore not portable to every setting.
Work through it
- Define the intended population and reference labels with qualified domain reviewers, separating participants across training and evaluation to avoid repeated-person leakage.
- Report sensitivity, specificity, precision and calibration with uncertainty on appropriate held-out data, examining population and measurement differences rather than changing prevalence mechanically.
- Compare thresholds only under a stated research decision framework and validate in the target setting before making any clinical-use claim.
Tradeoffs to discuss
- A threshold can trade missed outcomes against false positives, but acceptable costs require domain-specific judgment and evidence.
Common pitfalls
- Do not present a synthetic prevalence adjustment as prospective clinical validation.
- Avoid selecting a threshold on the same data used for its final performance report.