ORIGINAL COMPANY-CONTEXT PRACTICE
Evaluate an on-device classifier under resource limits
Original practice scenario: A hypothetical mobile feature must classify user-selected images locally. How would you choose a smaller model without assuming that average accuracy captures the user experience?
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
Evaluate task quality together with latency, memory and energy constraints on representative devices. Local execution changes deployment constraints but does not establish that a model is accurate or private by itself.
Work through it
- Create consented or synthetic evaluation data covering lighting, orientation, image quality and important classes; keep a separate held-out set.
- Compare candidate models and compression settings on the same cases, recording per-class errors and uncertainty alongside cold and warm inference latency.
- Test older supported hardware, thermal throttling, interrupted execution and unsupported inputs; define a fallback or clear inability response.
Tradeoffs to discuss
- Quantization or smaller architectures may reduce memory and energy use while disproportionately hurting rare classes.
Common pitfalls
- Avoid tuning repeatedly on the final evaluation set.
- Ensure diagnostics do not accidentally upload image content merely because inference itself runs on the device.