ORIGINAL TECHNICAL PRACTICE
Reduce AI latency with a measurable budget
A retrieval assistant misses its response-time target. How would you optimize it while preserving answer quality?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Break end-to-end latency into retrieval, reranking, context preparation, model first-token delay and generation. Optimize the measured bottleneck against a quality threshold rather than assuming a smaller model solves everything.
Work through it
- Record percentile latency by task type and request size. Separate a slow initial load from sustained serving performance and distinguish first-token time from total answer time.
- Test shorter contexts, better filtering, caching, fewer sequential tool calls and appropriate model selection. Evaluate the same task set after each change.
- Include throughput, failure rate and resource cost at realistic concurrency. Define a useful fallback when a dependency misses its deadline.
Tradeoffs to discuss
- Caching risks stale or incorrectly permission-scoped results. Streaming can improve perceived responsiveness while leaving total computation and answer correctness unchanged.
Common pitfalls
- Average latency hides tail failures. Removing evidence or shortening answers may meet a latency goal by silently weakening the task contract.