ORIGINAL COMPANY-CONTEXT PRACTICE
Admit inference jobs without exhausting accelerator memory
Original practice scenario: Design a hypothetical GPU inference service that mixes short interactive requests with large batch jobs. How would you admit work and handle uncertain memory estimates?
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
Treat memory as an admission constraint and latency as a separate scheduling objective. A queue alone cannot prevent several individually valid jobs from exhausting the same accelerator.
Work through it
- Estimate each job using model weights, batch size and expected working memory, reserving an explicit safety margin before dispatch.
- Maintain per-device reservations and separate interactive and batch queues; enforce tenant quotas and age waiting jobs to prevent starvation.
- Measure actual peak use, update estimates conservatively, and retry recoverable failures with a smaller batch or another device using a stable job identity.
Tradeoffs to discuss
- Conservative reservations lower utilization; aggressive packing improves throughput but increases retries and latency variance.
Common pitfalls
- Do not equate free memory observed before dispatch with capacity guaranteed during execution.
- Account for cancellation, failed workers and released reservations so capacity neither leaks nor becomes double-booked.