I am building a KV system that has code like:
async fn get_key(key: String, timeout: Duration)
pingora_timeout(timeout):
openraft.ensure_linearizable(ReadIndex).await;
return statemachine.read(key);
async fn write_key(key: String, value: String, timeout: Duration)
pingora_timeout(timeout):
openraft.client_write(key, value);
We are using ReadIndex because our env does not have an SLA on clock skew, and it would send heartbeats to peers in each call.
Assume heartbeat_interval is 50ms and timeout for the whole request is 500ms. For write_key, we can use the whole 500ms timeout budget, because if an append_entry call fails, openraft would automatically handle the retry. As long as the retry reaches quorum within 500ms the write is successful. However, get_key would return not enough for a quorum error without using the full 500ms budget. ensure_linearizable(ReadIndex) would not retry, it would send out heartbeats once and see if it can reach quorum. It can error out if the peers did not respond within 50ms due to a network gliche.
Without openraft change, there are two ways I can think of to work around the issue:
- retry the whole
ensure_linearizable call. This is the most straightforward way, but it may cause retry storm when some node is actually down. From the error message of ensure_linearizable, we could not tell if the error is retryable, i.e. we cannot tell if a peer is temporarily down or not.
- retry within
append_entry call. For non-heartbeat calls, we can let openraft handle retry as normal. But for heartbeat calls, we handle the retry ourselves. For example, if heartbeat_interval is 50ms, instead of making one RPC with 50ms timeout, we make an RPC call with 10ms instead and we can retry up to 5 times within the deadline. In this way, we do not need to retry on healthy node and we can stop retry if the RPC error indicate it is not retryable.
Otherwise we can modify openraft and the solution on top of my head is:
add a timeout option for ensure_linearizable, so it can retry heartbeats on the failed node until timeout is hit. I drafted a PR here with AI's help. But I am not sure if it will break raft correctness and whether openraft deliberately chose not to retry on heartbeats.
Can the community suggest what is the best way to work around of this issue?
I am building a KV system that has code like:
We are using
ReadIndexbecause our env does not have an SLA on clock skew, and it would send heartbeats to peers in each call.Assume
heartbeat_intervalis 50ms and timeout for the whole request is 500ms. Forwrite_key, we can use the whole 500ms timeout budget, because if anappend_entrycall fails, openraft would automatically handle the retry. As long as the retry reaches quorum within 500ms the write is successful. However,get_keywould returnnot enough for a quorumerror without using the full 500ms budget.ensure_linearizable(ReadIndex)would not retry, it would send out heartbeats once and see if it can reach quorum. It can error out if the peers did not respond within 50ms due to a network gliche.Without openraft change, there are two ways I can think of to work around the issue:
ensure_linearizablecall. This is the most straightforward way, but it may cause retry storm when some node is actually down. From the error message ofensure_linearizable, we could not tell if the error is retryable, i.e. we cannot tell if a peer is temporarily down or not.append_entrycall. For non-heartbeat calls, we can let openraft handle retry as normal. But for heartbeat calls, we handle the retry ourselves. For example, ifheartbeat_intervalis 50ms, instead of making one RPC with 50ms timeout, we make an RPC call with 10ms instead and we can retry up to 5 times within the deadline. In this way, we do not need to retry on healthy node and we can stop retry if the RPC error indicate it is not retryable.Otherwise we can modify openraft and the solution on top of my head is:
add a timeout option for
ensure_linearizable, so it can retry heartbeats on the failed node until timeout is hit. I drafted a PR here with AI's help. But I am not sure if it will break raft correctness and whether openraft deliberately chose not to retry on heartbeats.Can the community suggest what is the best way to work around of this issue?