ORIGINAL TECHNICAL PRACTICE
Design retries that do not amplify an outage
A dependency begins timing out. What retry policy would you propose?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Use bounded retries only when the operation is safe to repeat, within an overall deadline and retry budget. Backoff with jitter reduces synchronized retry waves but does not make an overloaded dependency healthy.
Work through it
- Classify transient and permanent failures and inspect any Retry-After instruction. Set connection and request timeouts using the dependency’s observed latency and the caller’s budget.
- Cap attempts and elapsed time, apply randomized backoff, and ensure only the appropriate layer retries. Reconcile uncertain non-idempotent operations before repeating them.
- Add overload protection and expose attempt counts, exhausted budgets and dependency failure rates. Test a partial outage and a large group of clients failing together.
Tradeoffs to discuss
- Retries can hide brief failures but increase load and tail latency. Failing promptly or returning a degraded result may better preserve the wider system during sustained overload.
Common pitfalls
- Retrying independently at several service layers multiplies attempts. A timeout does not prove that a side effect did not occur.