-
Couldn't load subscription status.
- Fork 10.3k
Description
Name and Version
bitnami/redis 20.0.5
What architecture are you using?
amd64
What steps will reproduce the bug?
- run chart with useHostnames=false (due to DNS issues causing tilt mode)
- enable sentinel
- enable masterService
Are you using any custom parameters or values?
useHostnames: false
sentinel:
enabled: true
masterService:
enabled: trueWhat is the expected behavior?
master pod is labeled with isMaster="true"
What do you see instead?
no such label on master
Additional information
kubectl uses the content of /etc/shared/current for the pod selector in update-master-label.sh.
The content of /etc/shared/current is written by sentinel from ${REDIS_MASTER_HOST/.*} in start-sentinel.sh.
when using useHostnames=false REDIS_MASTER_HOST contains not redis-node-0.redis.svc.cluster.local, which would return redis-node-0 as current, but instead the IP address e.g. 172.22.140.140 which returns 172 as current.
thus kubectl matches no pod and updates no labels.
Unfortunately, the output of kubectl is also not logged - only the echo statement with new master elected, updating label(s)....
Patching the script configmap with echo "$(< "/etc/shared/current")" helped to debug this as it returned 172 instead of redis-node-0.
This also applies to the delete patch for the label app.kubernetes.io/role which will be fixed in 20.0.6 (annotation -> label).
To fix this either sentinel writes a non-resolved value for REDIS_MASTER_HOST into /etc/shared/current
or the kubectl script looks up the pod by IP if current is numerical. such as:
kubectl get -n <namespace> --output json pods | jq '.items[] | select(.status.podIP=="<IP>")' | jq .metadata.name
though this would likely require further RBAC permissions.