Original practice scenario: Design a hypothetical GPU inference service that mixes short interactive requests with large batch jobs. How would you admit work and handle uncertain memory estimates?
Read model answer +
Treat memory as an admission constraint and latency as a separate scheduling objective. A queue alone cannot prevent several individually valid jobs from exhausting the same accelerator.
Work through it
Estimate each job using model weights, batch size and expected working memory, reserving an explicit safety margin before dispatch.
Maintain per-device reservations and separate interactive and batch queues; enforce tenant quotas and age waiting jobs to prevent starvation.
Measure actual peak use, update estimates conservatively, and retry recoverable failures with a smaller batch or another device using a stable job identity.
Tradeoffs to discuss
Conservative reservations lower utilization; aggressive packing improves throughput but increases retries and latency variance.
Common pitfalls
Do not equate free memory observed before dispatch with capacity guaranteed during execution.
Account for cancellation, failed workers and released reservations so capacity neither leaks nor becomes double-booked.
Original practice scenario: A hypothetical engineering team has two GPU kernels and observes faster average execution for the new one. Design an evaluation that avoids a misleading speedup claim.
Read model answer +
Define the workload population and correctness tolerance before timing. A speedup is meaningful only for comparable outputs, hardware conditions and representative input shapes.
Work through it
Construct cases spanning small and large tensors, awkward dimensions, data types and memory-pressure conditions; compare outputs against an appropriate reference.
Separate compilation and warm-up from steady execution, synchronize device work before measuring, and randomize repeated kernel order.
Report latency distributions and memory use by workload slice, including regressions, rather than averaging unrelated cases into one impressive ratio.
Tradeoffs to discuss
A controlled benchmark isolates kernel effects but may omit contention and data-transfer costs present in an application.
Common pitfalls
Avoid timing only asynchronous launch overhead while the device continues working.
Do not discard slow or failed cases after seeing the results; disclose exclusions and evaluate end-to-end impact separately.
