ORIGINAL TECHNICAL PRACTICE
Choose precision and recall for the task
A rare-event detector has 99% accuracy. What would you ask before deciding whether it is useful?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Accuracy alone is uninformative without class prevalence and the costs of mistakes. Precision measures how many positive predictions are correct; recall measures how many actual positives were detected.
Work through it
- Inspect the confusion matrix and the number of positive examples. A detector predicting every item negative can achieve99% accuracy when positives make up1% of the data.
- Translate false positives and false negatives into operational consequences, such as reviewer workload and missed incidents. Choose a decision threshold using a validation set.
- Report precision, recall and relevant threshold curves with uncertainty and subgroup checks. Evaluate on a held-out sample reflecting the deployment population.
Tradeoffs to discuss
- Increasing recall often increases review workload by admitting more false positives. A combined score is useful only when its weighting fits the task.
Common pitfalls
- A metric calculated on artificially balanced data may not describe production precision. Choosing a threshold on the final test set leaks evaluation information.