ORIGINAL COMPANY-CONTEXT PRACTICE
Compare accelerator kernels fairly
Original practice scenario: A hypothetical engineering team has two GPU kernels and observes faster average execution for the new one. Design an evaluation that avoids a misleading speedup claim.
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
Define the workload population and correctness tolerance before timing. A speedup is meaningful only for comparable outputs, hardware conditions and representative input shapes.
Work through it
- Construct cases spanning small and large tensors, awkward dimensions, data types and memory-pressure conditions; compare outputs against an appropriate reference.
- Separate compilation and warm-up from steady execution, synchronize device work before measuring, and randomize repeated kernel order.
- Report latency distributions and memory use by workload slice, including regressions, rather than averaging unrelated cases into one impressive ratio.
Tradeoffs to discuss
- A controlled benchmark isolates kernel effects but may omit contention and data-transfer costs present in an application.
Common pitfalls
- Avoid timing only asynchronous launch overhead while the device continues working.
- Do not discard slow or failed cases after seeing the results; disclose exclusions and evaluate end-to-end impact separately.