[bitnami/redis] Shuts down the node where the master is located, the sentinels on the other two slave nodes will become stuck. #36305
Description
Name and Version
redis/20.2.0
What steps will reproduce the bug?
In a Redis 3-node sentinel mode setup, after the BMC shuts down the node where the master is located, the sentinels on the other two slave nodes will become stuck.
After the sentinel nodes recover from being stuck, using SENTINEL get-master-addr-by-name mymaster still returns the old master node.
Env:
- k8s v1.23.17
- Redis version=7.4.1
Sentinel pod event ：
2025-09-29T09:41:43Z 2025-09-29T09:41:28Z Warning Unhealthy openstack-redis-node-1 Readiness probe failed: command "sh -c /health/ping_sentinel.sh 1" timed out
2025-09-29T09:41:37Z 2025-09-29T09:41:37Z Warning Unhealthy openstack-redis-node-1 Liveness probe failed: command "sh -c /health/ping_sentinel.sh 5" timed out
Are you using any custom parameters or values?
no
What is the expected behavior?
1:X 29 Sep 2025 09:42:25.385 # Failed to resolve hostname 'openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local'
1:X 29 Sep 2025 09:42:26.433 # Failed to resolve hostname 'openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local'
1:X 29 Sep 2025 09:42:27.428 # -tilt #tilt mode exited
1:X 29 Sep 2025 09:42:27.428 # +sdown master mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:27.428 # +sdown sentinel 3acccca4a648813efac4e831aaa9409aa70d6d25 openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 26379 @ mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:27.461 # Failed to resolve hostname 'openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local'
1:X 29 Sep 2025 09:42:27.506 # Failed to resolve hostname 'openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local'
1:X 29 Sep 2025 09:42:27.506 # +odown master mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379 #quorum 2/2
1:X 29 Sep 2025 09:42:27.506 # +new-epoch 10
1:X 29 Sep 2025 09:42:27.506 # +try-failover master mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:27.507 * Sentinel new configuration saved on disk
1:X 29 Sep 2025 09:42:27.507 # +vote-for-leader 2000720065fc8dcb5a92ec0b58128a96b9170133 10
1:X 29 Sep 2025 09:42:27.508 # Failed to resolve hostname 'openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local'
1:X 29 Sep 2025 09:42:27.510 * 84b4093eae708d0d36e4bafdb7e0a5c297ebae17 voted for 2000720065fc8dcb5a92ec0b58128a96b9170133 10
1:X 29 Sep 2025 09:42:27.571 # +elected-leader master mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:27.571 # +failover-state-select-slave master mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:27.672 # +selected-slave slave openstack-redis-node-0.openstack-redis-headless.openstack.svc.cluster.local:6379 openstack-redis-node-0.openstack-redis-headless.openstack.svc.cluster.local 6379 @ mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:27.672 * +failover-state-send-slaveof-noone slave openstack-redis-node-0.openstack-redis-headless.openstack.svc.cluster.local:6379 openstack-redis-node-0.openstack-redis-headless.openstack.svc.cluster.local 6379 @ mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:27.727 * +failover-state-wait-promotion slave openstack-redis-node-0.openstack-redis-headless.openstack.svc.cluster.local:6379 openstack-redis-node-0.openstack-redis-headless.openstack.svc.cluster.local 6379 @ mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:28.430 * Sentinel new configuration saved on disk
1:X 29 Sep 2025 09:42:28.430 # +promoted-slave slave openstack-redis-node-0.openstack-redis-headless.openstack.svc.cluster.local:6379 openstack-redis-node-0.openstack-redis-headless.openstack.svc.cluster.local 6379 @ mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:28.430 # +failover-state-reconf-slaves master mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:28.477 # Failed to resolve hostname 'openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local'
1:X 29 Sep 2025 09:42:28.485 # +failover-end master mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379
1:X 29 Sep 2025 09:42:28.485 # +switch-master mymaster openstack-redis-node-2.openstack-redis-headless.openstack.svc.cluster.local 6379 openstack-redis-node-0.openstack-redis-headless.openstack.svc.cluster.local 6379
What do you see instead?
The sentinels on the other two slave nodes will become stuck.