rustfs high availability

[jurim76]

Cluster setup: 4 nodes with ec:2 + pgbackup bucket, running in k8s cluster

k get pod
rustfs-0                                   1/1     Running   3 (19h ago)   19h
rustfs-1                                   1/1     Running   3 (19h ago)   19h
rustfs-2                                   1/1     Running   3 (47m ago)   47m
rustfs-3                                   1/1     Running   3 (19h ago)   19h

• Test bucket availability with mc pod

k exec -it mclient -- bash
$ mc ls rustfs/pgbackup/
[2025-12-24 12:24:09 UTC]     0B test/

• one worker node was powered off (rustfs-2 pod)
• mc got a the huge delay to access the bucket (~30sec)
• the pods are still alive, the pod terminated by kubelet after the timeout (~5 min)

rustfs-0                                   1/1     Running       3 (19h ago)   19h
rustfs-1                                   1/1     Running       3 (19h ago)   19h
rustfs-2                                   1/1     Terminating   3 (57m ago)   57m
rustfs-3                                   1/1     Running       3 (19h ago)   19h

In this scenario rustfs couldn't provide the HA due a huge delay (~30sec) for client
Please share your thoughts about it, perhaps my setup is wrong.

[majinghe]

@jurim76 I think this is not about RustFS HA, just about kubernetes pod eviction policy; generally, pod-eviction-timeout is 5 mins. Once node powers off, but pod can't be deleted and scheduled to a new node. Pod eviction policy has their own policy. tolerations maybe helpful, but the timeout setting should be tested.

[jurim76]

@majinghe
The pod couldn't be scheduled on the other node due a volume affinity conflict.
Even the pod marked as terminated and pod's ip was removed from rustfs-svc endpoint, the client still gets a delay
I guess the rustfs operator could provide the HA with nodes health check and implementing auto-failover mechanism, but operator should manage the pods directly, without statefulset limitations.

[majinghe]

@jurim76 Yes, that's what operator do and that's what big difference between helm and operator. RustFS operator is under development, will take a long time to GA.