ORIGINAL TECHNICAL PRACTICE
Make a recovery plan testable
How would you design and verify recovery for a stateful service after a region becomes unavailable?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Start with recovery-time and recovery-point objectives, then identify the data, configuration and external dependencies needed to resume the service. A backup is useful only if it can be restored correctly.
Work through it
- Choose replication and backup strategies matching tolerated data loss. Include encryption keys, schema versions and infrastructure configuration in the recovery plan.
- Define failover authority, fencing of old writers and traffic routing. Account for delayed messages and duplicate operations when service resumes.
- Run restoration and failover exercises using representative data. Measure actual recovery time, verify invariants and document how to reconcile changes made during an incident.
Tradeoffs to discuss
- Active-active systems can improve availability while making conflict resolution harder. A simpler active-passive design may be easier to operate under a looser recovery objective.
Common pitfalls
- Replicating a deletion quickly is not the same as retaining a recoverable backup. Failing over without fencing can permit two regions to accept incompatible authoritative writes.