Allow users to guarantee redundancy of keys migrated during slot migrations #2723
Description
Problem Statement
During normal operation where slot assignments are kept stable, users can use the WAIT command to ensure keys they add are persisted on a given number of replicas.
In the existing slot migration implementations, there is no way to specify such guarantees:
- Atomic slot migration: keys replicated to the target shard during the migration are not guaranteed to be persisted on replica nodes when the slot ownership is transferred
- Legacy slot migration: after calling
MIGRATE, the key is not guaranteed to be persisted to replicas before it is deleted on the source shard.
Proposed Solution
Atomic slot migration
Provide the ability to specify a minimum number of replicas in the CLUSTER MIGRATESLOTS command. Before the slot migration target sends the CLUSTER SYNCSLOTS REQUEST-FAILOVER command, the target shard would first request and wait for REPLCONF ACKs from replicas to meet the WAIT condition.
CLUSTER MIGRATESLOTS SLOTSRANGE <start> <end> NODE <node-id> WAITREPLICAS <num>
Legacy slot migration
Similar to above, but specify this in each MIGRATE command. Operators could send this on each MIGRATE command, or only on those affecting keys that are sensitive.
MIGRATE host port 〈 key | "" 〉 destination-db timeout [ COPY ] [ REPLACE ] [ AUTH auth | AUTH2 username password ] [ WAIT numreplicas timeout ] [ KEYS keys ] [ [ KEYS keys ] ... ]
Other thoughts
Personally, I don't know how much we should work on plugging holes through the existing WAIT-style system, or if we should invest our efforts in #1355. But this was something brought up during the atomic slot migration design review so I wanted to make sure we had this discussion.