make CNV fails while running configure-network.sh because keepalived VIPs are lost and the api becomes unreachable

[mcornea]

The API VIP is lost after the CNV/configure-network.sh scripts gets run:

11:15:01 TASK [Run CNV make] ************************************************************
echo 'Creating registry secret'", "+ cat", "+ oc create -f -", "+ echo 'Creating OperatorGroup'", "+ cat", "+ oc create -f -", "+ echo 'Creating OperatorSource'", "+ cat", "+ oc create -f -", "+ echo 'Give the cluster 30 seconds to create the catalogSourceConfig...'", "+ sleep 30", "+ cat", "+ oc apply -f -", "+ echo 'Give the cluster 30 seconds to process catalogSourceConfig...'", "+ sleep 30", "+ oc wait deploy hco-catalogsource-config --for condition=available -n openshift-marketplace --timeout=360s", "++ seq 1 10", "+ for i in $(seq 1 $RETRIES)", "+ echo 'Waiting for packagemanifest '\''kubevirt-hyperconverged'\'' to be created in namespace '\''openshift-cnv'\''...'", "+ oc get packagemanifest -n openshift-cnv kubevirt-hyperconverged", "+ break", "+ echo 'Creating Subscription'", "+ cat", "+ oc create -f -", "+ echo 'Give OLM 60 seconds to process the subscription...'", "+ sleep 60", "+ sed 's/approved: false/approved: true/'", "+ oc apply -n openshift-cnv -f -", "++ oc get installplan -n openshift-cnv --no-headers", "++ grep kubevirt-hyperconverged-operator.v2.1.0", "++ awk '{print $1}'", "+ oc get installplan -o yaml -n openshift-cnv install-d8857", "Warning: oc apply should be used on resource created by either oc create --save-config or oc apply", "+ echo 'Give OLM 60 seconds to process the installplan...'", "+ sleep 60", "++ oc get pods -n openshift-cnv", "++ grep hco-operator", "++ head -1", "++ awk '{ print $1 }'", "+ oc wait pod hco-operator-76f448c546-z7tl4 --for condition=Ready -n openshift-cnv --timeout=360s", "+ echo 'Creating the HCO'\''s Custom Resource'", "+ cat", "+ oc create -f -", "+ echo 'Waiting for HCO to get fully deployed'", "+ oc wait -n openshift-cnv hyperconverged hyperconverged-cluster --for condition=Available --timeout=10m", "++ grep machineCIDR ../OpenShift/install-config.yaml", "++ sed 's/\(.\): \(.\)/\2/'", "+ MACHINE_CIDR=192.168.123.0/24", "+ BRIDGE_NAME=brext", "+ export KUBECONFIG=../OpenShift/ocp/auth/kubeconfig", "+ KUBECONFIG=../OpenShift/ocp/auth/kubeconfig", "++ oc get nodes -o 'jsonpath={range .items[]}{.metadata.name} {end}'", "+ nodes='rhhi-node-master-0 rhhi-node-master-1 rhhi-node-master-2 '", "+ echo 'Configuring networks on nodes'", "+ for node in $nodes", "+ echo 'Detecting the default interface'", "++ oc get nodenetworkstate rhhi-node-master-0 -o 'jsonpath={.status.currentState.routes.running[?(@.destination=="192.168.123.0/24")].next-hop-interface}'", "++ cut -d ' ' -f 1", "+ default_iface=ens4", "+ '[' ens4 == brext ']'", "+ echo 'Detecting MAC address of the default interface'", "++ oc get nodenetworkstate rhhi-node-master-0 -o 'jsonpath={.status.currentState.interfaces[?(@.name=="ens4")].mac-address}'", "+ default_iface_mac=52:54:00:DD:42:24", "+ echo 'Applying node network configuration policy'", "+ cat", "+ oc apply -f -", "+ for node in $nodes", "+ echo 'Detecting the default interface'", "++ oc get nodenetworkstate rhhi-node-master-1 -o 'jsonpath={.status.currentState.routes.running[?(@.destination=="192.168.123.0/24")].next-hop-interface}'", "++ cut -d ' ' -f 1", "+ default_iface=ens4", "+ '[' ens4 == brext ']'", "+ echo 'Detecting MAC address of the default interface'", "++ oc get nodenetworkstate rhhi-node-master-1 -o 'jsonpath={.status.currentState.interfaces[?(@.name=="ens4")].mac-address}'", "+ default_iface_mac=52:54:00:BF:3F:46", "+ echo 'Applying node network configuration policy'", "+ cat", "+ oc apply -f -", "+ for node in $nodes", "+ echo 'Detecting the default interface'", "++ oc get nodenetworkstate rhhi-node-master-2 -o 'jsonpath={.status.currentState.routes.running[?(@.destination=="192.168.123.0/24")].next-hop-interface}'", "++ cut -d ' ' -f 1", "+ default_iface=ens4", "+ '[' ens4 == brext ']'", "+ echo 'Detecting MAC address of the default interface'", "++ oc get nodenetworkstate rhhi-node-master-2 -o 'jsonpath={.status.currentState.interfaces[?(@.name=="ens4")].mac-address}'", "+ default_iface_mac=52:54:00:B7:79:20", "+ echo 'Applying node network configuration policy'", "+ cat", "+ oc apply -f -", "+ echo 'Waiting until the configuration is done, it may take up to 5 minutes until keepalived gets reconfigured'", "+ for node in $nodes", "++ oc get nodenetworkstate rhhi-node-master-0 -o 'jsonpath={.status.currentState.routes.running[?(@.destination=="192.168.123.0/24")].next-hop-interface}'", "+ '[' 'ens4 ens4' == brext ']'", "+ sleep 10", "++ oc get nodenetworkstate rhhi-node-master-0 -o 'jsonpath={.status.currentState.routes.running[?(@.destination=="192.168.123.0/24")].next-hop-interface}'", "+ '[' brext == brext ']'", "+ oc wait node rhhi-node-master-0 --for condition=Ready --timeout=10m", "Unable to connect to the server: dial tcp 192.168.123.5:6443: connect: no route to host", "make[1]: *** [Makefile:9: deploy] Error 1", "make: *** [Makefile:25: CNV] Error 2"], "stdout": "", "stdout_lines": []}

Checking brext bridges on the master nodes none of them has the API VIP:

[kni@rhhi-node-worker-0 install-scripts]$ for master in {0..2}; do ssh core@rhhi-node-master-$master 'ip a s dev brext';done
71: brext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:dd:42:24 brd ff:ff:ff:ff:ff:ff
    inet 192.168.123.126/24 brd 192.168.123.255 scope global dynamic noprefixroute brext
       valid_lft 2929sec preferred_lft 2929sec
    inet6 fe80::97a2:f95:f7da:a30a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
61: brext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:bf:3f:46 brd ff:ff:ff:ff:ff:ff
    inet 192.168.123.130/24 brd 192.168.123.255 scope global dynamic noprefixroute brext
       valid_lft 2928sec preferred_lft 2928sec
    inet6 fe80::fb17:8f64:42ab:5263/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
70: brext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:b7:79:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.123.114/24 brd 192.168.123.255 scope global dynamic noprefixroute brext
       valid_lft 2931sec preferred_lft 2931sec
    inet6 fe80::d0a2:6ba6:24ea:514a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[mcornea]

/cc @phoracek

[phoracek]

@mcornea the OpenShift release may be outdated, @celebdor pushed some changes to machineconfig and MCO to fix keepalived. On your setup, it is likely outdated and does not configure keepalived IPs on the new default interface.

[mcornea]

@mcornea the OpenShift release may be outdated, @celebdor pushed some changes to machineconfig and MCO to fix keepalived. On your setup, it is likely outdated and does not configure keepalived IPs on the new default interface.

I'm using the version provided by a fresh deployment with install-scripts. How can we update the version to get the deployment back in a working state?

[celebdor]

Check please with crictl if you have some keepalived-monitor container

[mcornea]

Check please with crictl if you have some keepalived-monitor container

It is there:

[kni@rhhi-node-worker-0 ~]$ for master in {0..2}; do ssh core@rhhi-node-master-$master 'sudo crictl ps | grep keepalived-monitor';done
dd5a6585e9133       32c724f01ed6fafe3daf62eabd0151413696e27229bd7f59e0ca749fe0d529ce                                                                                                                               About an hour ago   Running             keepalived-monitor                      1                   4d3d37d44d982
cea8db649fa84       32c724f01ed6fafe3daf62eabd0151413696e27229bd7f59e0ca749fe0d529ce                                                                                                                               About an hour ago   Running             keepalived-monitor                      1                   ad4f8e58aa58f
1abadaa14931e       32c724f01ed6fafe3daf62eabd0151413696e27229bd7f59e0ca749fe0d529ce                                                                                                                               About an hour ago   Running             keepalived-monitor                        1                   6a637becec22d

[mcornea]

Logs from one of the keepalived-monitor containers:

Note that 192.168.123.1 is the default router, there are no etcd records configured there.

[root@rhhi-node-master-1 core]# cat /etc/resolv.conf 
# Generated by NetworkManager
search rhhi-virt-cluster.qe.lab.redhat.com
nameserver 192.168.123.6
nameserver 192.168.123.1

[root@rhhi-node-master-1 core]# crictl logs -f cea8db649fa84
time="2019-09-23T15:07:48Z" level=info msg="Config change detected" new config="{{rhhi-virt-cluster qe.lab.redhat.com 192.168.123.5 99 192.168.123.6 94 192.168.123.10 68 24 0} {0 0 0 [{etcd-2.rhhi-virt-cluster.qe.lab.redhat.com. 192.168.123.114 0} {etcd-0.rhhi-virt-cluster.qe.lab.redhat.com. 192.168.123.126 0}]} 192.168.123.130 rhhi-node-master-1 rhhi-node-etcd-1 ens4 [192.168.123.1]}"
time="2019-09-23T15:07:48Z" level=info msg="Runtimecfg rendering template" path=/etc/keepalived/keepalived.conf
time="2019-09-23T15:21:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:21:59Z" level=info msg="Config change detected" new config="{{rhhi-virt-cluster qe.lab.redhat.com 192.168.123.5 99 192.168.123.6 94 192.168.123.10 68 24 0} {0 0 0 []} 192.168.123.130 rhhi-node-master-1 rhhi-node-etcd-1 brext [192.168.123.1]}"
time="2019-09-23T15:21:59Z" level=info msg="Runtimecfg rendering template" path=/etc/keepalived/keepalived.conf
time="2019-09-23T15:22:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:22:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:22:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:23:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:23:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:23:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:24:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:24:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:24:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:25:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:25:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:25:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:26:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:26:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:26:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:27:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:27:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:27:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:28:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:28:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:28:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:29:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:29:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:29:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:30:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:30:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:30:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:31:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:31:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:31:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:32:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:32:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:32:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:33:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:33:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:33:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:34:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:34:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:34:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:35:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:35:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:35:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:36:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:36:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:36:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:37:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:37:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:37:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:38:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:38:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:38:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:39:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:39:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:39:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:40:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:40:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:40:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:41:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:41:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:41:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:42:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:42:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:42:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:43:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:43:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:43:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:44:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:44:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:44:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:45:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:45:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:45:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:46:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:46:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:46:59Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:47:19Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:47:39Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:48:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:48:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:48:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:49:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:49:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:49:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:50:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:50:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:50:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:51:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:51:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:51:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:52:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:52:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:52:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:53:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:53:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:53:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:54:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:54:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:54:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:55:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:55:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:55:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:56:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:56:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:56:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:57:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:57:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:57:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:58:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:58:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:58:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:59:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:59:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T15:59:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T16:00:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T16:00:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T16:00:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T16:01:00Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T16:01:20Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"
time="2019-09-23T16:01:40Z" level=info msg="Failed to get Etcd SRV members" err="lookup _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com on 192.168.123.1:53: no such host"

[celebdor]

Apart from the DNS stuff, which should not affect the keepalived functionality, it seems the change was detected. Can you confirm that all masters have the IPs on brext and that /etc/keepalived/keepalived.conf points to the brext?

[celebdor]

Are those master IPs correct? The same as before installing CNV?

[mcornea]

Are those master IPs correct? The same as before installing CNV?

Yes, IPs look correct:

kni@rhhi-node-worker-0 ~]$ for master in {0..2}; do dig +short rhhi-node-master-$master; ssh core@rhhi-node-master-$master 'ip a s dev brext; sudo cat /etc/keepalived/keepalived.conf'; echo '########################';done
192.168.123.126
71: brext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:dd:42:24 brd ff:ff:ff:ff:ff:ff
    inet 192.168.123.126/24 brd 192.168.123.255 scope global dynamic noprefixroute brext
       valid_lft 3098sec preferred_lft 3098sec
    inet6 fe80::97a2:f95:f7da:a30a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
vrrp_script chk_ocp {
    script "/usr/bin/curl -o /dev/null -kLs https://0:6443/readyz"
    interval 1
    weight 50
}

vrrp_script chk_dns {
    script "/usr/bin/host -t SRV _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com localhost"
    interval 1
    weight 50
}

# TODO: Improve this check. The port is assumed to be alive.
# Need to assess what is the ramification if the port is not there.
vrrp_script chk_ingress {
    script "/usr/bin/curl -o /dev/null -kLs http://0:1936/healthz"
    interval 1
    weight 50
}

vrrp_instance rhhi-virt-cluster_API {
    state BACKUP
    interface brext
    virtual_router_id 99
    priority 40
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass rhhi-virt-cluster_api_vip
    }
    virtual_ipaddress {
        192.168.123.5/24
    }
    track_script {
        chk_ocp
    }
}

vrrp_instance rhhi-virt-cluster_DNS {
    state BACKUP
    interface brext
    virtual_router_id 94
    priority 40
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass rhhi-virt-cluster_dns_vip
    }
    virtual_ipaddress {
        192.168.123.6/24
    }
    track_script {
        chk_dns
    }
}

vrrp_instance rhhi-virt-cluster_INGRESS {
    state BACKUP
    interface brext
    virtual_router_id 68
    priority 40
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass cluster_uuid_ingress_vip
    }
    virtual_ipaddress {
        192.168.123.10/24
    }
    track_script {
        chk_ingress
    }
}
########################
192.168.123.130
61: brext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:bf:3f:46 brd ff:ff:ff:ff:ff:ff
    inet 192.168.123.130/24 brd 192.168.123.255 scope global dynamic noprefixroute brext
       valid_lft 3182sec preferred_lft 3182sec
    inet6 fe80::fb17:8f64:42ab:5263/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
vrrp_script chk_ocp {
    script "/usr/bin/curl -o /dev/null -kLs https://0:6443/readyz"
    interval 1
    weight 50
}

vrrp_script chk_dns {
    script "/usr/bin/host -t SRV _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com localhost"
    interval 1
    weight 50
}

# TODO: Improve this check. The port is assumed to be alive.
# Need to assess what is the ramification if the port is not there.
vrrp_script chk_ingress {
    script "/usr/bin/curl -o /dev/null -kLs http://0:1936/healthz"
    interval 1
    weight 50
}

vrrp_instance rhhi-virt-cluster_API {
    state BACKUP
    interface brext
    virtual_router_id 99
    priority 40
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass rhhi-virt-cluster_api_vip
    }
    virtual_ipaddress {
        192.168.123.5/24
    }
    track_script {
        chk_ocp
    }
}

vrrp_instance rhhi-virt-cluster_DNS {
    state BACKUP
    interface brext
    virtual_router_id 94
    priority 40
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass rhhi-virt-cluster_dns_vip
    }
    virtual_ipaddress {
        192.168.123.6/24
    }
    track_script {
        chk_dns
    }
}

vrrp_instance rhhi-virt-cluster_INGRESS {
    state BACKUP
    interface brext
    virtual_router_id 68
    priority 40
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass cluster_uuid_ingress_vip
    }
    virtual_ipaddress {
        192.168.123.10/24
    }
    track_script {
        chk_ingress
    }
}
########################
192.168.123.114
70: brext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:b7:79:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.123.114/24 brd 192.168.123.255 scope global dynamic noprefixroute brext
       valid_lft 3229sec preferred_lft 3229sec
    inet6 fe80::d0a2:6ba6:24ea:514a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
vrrp_script chk_ocp {
    script "/usr/bin/curl -o /dev/null -kLs https://0:6443/readyz"
    interval 1
    weight 50
}

vrrp_script chk_dns {
    script "/usr/bin/host -t SRV _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com localhost"
    interval 1
    weight 50
}

# TODO: Improve this check. The port is assumed to be alive.
# Need to assess what is the ramification if the port is not there.
vrrp_script chk_ingress {
    script "/usr/bin/curl -o /dev/null -kLs http://0:1936/healthz"
    interval 1
    weight 50
}

vrrp_instance rhhi-virt-cluster_API {
    state BACKUP
    interface brext
    virtual_router_id 99
    priority 40
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass rhhi-virt-cluster_api_vip
    }
    virtual_ipaddress {
        192.168.123.5/24
    }
    track_script {
        chk_ocp
    }
}

vrrp_instance rhhi-virt-cluster_DNS {
    state BACKUP
    interface brext
    virtual_router_id 94
    priority 40
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass rhhi-virt-cluster_dns_vip
    }
    virtual_ipaddress {
        192.168.123.6/24
    }
    track_script {
        chk_dns
    }
}

vrrp_instance rhhi-virt-cluster_INGRESS {
    state BACKUP
    interface brext
    virtual_router_id 68
    priority 40
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass cluster_uuid_ingress_vip
    }
    virtual_ipaddress {
        192.168.123.10/24
    }
    track_script {
        chk_ingress
    }
}
########################

[celebdor]

Can we have the logs for the keepalived containers?

…

On Mon, Sep 23, 2019, 18:19 Marius Cornea ***@***.***> wrote: Are those master IPs correct? The same as before installing CNV? Yes, IPs look correct: ***@***.*** ~]$ for master in {0..2}; do dig +short rhhi-node-master-$master; ssh ***@***.***$master 'ip a s dev brext; sudo cat /etc/keepalived/keepalived.conf'; echo '########################';done 192.168.123.126 71: brext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:dd:42:24 brd ff:ff:ff:ff:ff:ff inet 192.168.123.126/24 brd 192.168.123.255 scope global dynamic noprefixroute brext valid_lft 3098sec preferred_lft 3098sec inet6 fe80::97a2:f95:f7da:a30a/64 scope link noprefixroute valid_lft forever preferred_lft forever vrrp_script chk_ocp { script "/usr/bin/curl -o /dev/null -kLs https://0:6443/readyz" interval 1 weight 50 } vrrp_script chk_dns { script "/usr/bin/host -t SRV _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com localhost" interval 1 weight 50 } # TODO: Improve this check. The port is assumed to be alive.# Need to assess what is the ramification if the port is not there. vrrp_script chk_ingress { script "/usr/bin/curl -o /dev/null -kLs http://0:1936/healthz" interval 1 weight 50 } vrrp_instance rhhi-virt-cluster_API { state BACKUP interface brext virtual_router_id 99 priority 40 advert_int 1 authentication { auth_type PASS auth_pass rhhi-virt-cluster_api_vip } virtual_ipaddress { 192.168.123.5/24 } track_script { chk_ocp } } vrrp_instance rhhi-virt-cluster_DNS { state BACKUP interface brext virtual_router_id 94 priority 40 advert_int 1 authentication { auth_type PASS auth_pass rhhi-virt-cluster_dns_vip } virtual_ipaddress { 192.168.123.6/24 } track_script { chk_dns } } vrrp_instance rhhi-virt-cluster_INGRESS { state BACKUP interface brext virtual_router_id 68 priority 40 advert_int 1 authentication { auth_type PASS auth_pass cluster_uuid_ingress_vip } virtual_ipaddress { 192.168.123.10/24 } track_script { chk_ingress } }######################## 192.168.123.130 61: brext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:bf:3f:46 brd ff:ff:ff:ff:ff:ff inet 192.168.123.130/24 brd 192.168.123.255 scope global dynamic noprefixroute brext valid_lft 3182sec preferred_lft 3182sec inet6 fe80::fb17:8f64:42ab:5263/64 scope link noprefixroute valid_lft forever preferred_lft forever vrrp_script chk_ocp { script "/usr/bin/curl -o /dev/null -kLs https://0:6443/readyz" interval 1 weight 50 } vrrp_script chk_dns { script "/usr/bin/host -t SRV _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com localhost" interval 1 weight 50 } # TODO: Improve this check. The port is assumed to be alive.# Need to assess what is the ramification if the port is not there. vrrp_script chk_ingress { script "/usr/bin/curl -o /dev/null -kLs http://0:1936/healthz" interval 1 weight 50 } vrrp_instance rhhi-virt-cluster_API { state BACKUP interface brext virtual_router_id 99 priority 40 advert_int 1 authentication { auth_type PASS auth_pass rhhi-virt-cluster_api_vip } virtual_ipaddress { 192.168.123.5/24 } track_script { chk_ocp } } vrrp_instance rhhi-virt-cluster_DNS { state BACKUP interface brext virtual_router_id 94 priority 40 advert_int 1 authentication { auth_type PASS auth_pass rhhi-virt-cluster_dns_vip } virtual_ipaddress { 192.168.123.6/24 } track_script { chk_dns } } vrrp_instance rhhi-virt-cluster_INGRESS { state BACKUP interface brext virtual_router_id 68 priority 40 advert_int 1 authentication { auth_type PASS auth_pass cluster_uuid_ingress_vip } virtual_ipaddress { 192.168.123.10/24 } track_script { chk_ingress } }######################## 192.168.123.114 70: brext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:b7:79:20 brd ff:ff:ff:ff:ff:ff inet 192.168.123.114/24 brd 192.168.123.255 scope global dynamic noprefixroute brext valid_lft 3229sec preferred_lft 3229sec inet6 fe80::d0a2:6ba6:24ea:514a/64 scope link noprefixroute valid_lft forever preferred_lft forever vrrp_script chk_ocp { script "/usr/bin/curl -o /dev/null -kLs https://0:6443/readyz" interval 1 weight 50 } vrrp_script chk_dns { script "/usr/bin/host -t SRV _etcd-server-ssl._tcp.rhhi-virt-cluster.qe.lab.redhat.com localhost" interval 1 weight 50 } # TODO: Improve this check. The port is assumed to be alive.# Need to assess what is the ramification if the port is not there. vrrp_script chk_ingress { script "/usr/bin/curl -o /dev/null -kLs http://0:1936/healthz" interval 1 weight 50 } vrrp_instance rhhi-virt-cluster_API { state BACKUP interface brext virtual_router_id 99 priority 40 advert_int 1 authentication { auth_type PASS auth_pass rhhi-virt-cluster_api_vip } virtual_ipaddress { 192.168.123.5/24 } track_script { chk_ocp } } vrrp_instance rhhi-virt-cluster_DNS { state BACKUP interface brext virtual_router_id 94 priority 40 advert_int 1 authentication { auth_type PASS auth_pass rhhi-virt-cluster_dns_vip } virtual_ipaddress { 192.168.123.6/24 } track_script { chk_dns } } vrrp_instance rhhi-virt-cluster_INGRESS { state BACKUP interface brext virtual_router_id 68 priority 40 advert_int 1 authentication { auth_type PASS auth_pass cluster_uuid_ingress_vip } virtual_ipaddress { 192.168.123.10/24 } track_script { chk_ingress } }######################## — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#160?email_source=notifications&email_token=AAEXHH5TKA6NJHCGLEH3SZTQLDUA5A5CNFSM4IZMNOIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7LNQKI#issuecomment-534173737>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEXHH5UE6KPKGS7JHJF42DQLDUA5ANCNFSM4IZMNOIA> .

[mcornea]

keepalived-rhhi-node-master-0.log
keepalived-rhhi-node-master-1.log
keepalived-rhhi-node-master-2.log

[celebdor]

The reload and reconfiguration happened. For some reason the health checks all fail though. I would need access to one such deployment.

[celebdor]

curl: (7) Failed to connect to 0 port 6443: Connection refused```

[qinqon]

I have another env with same issues, as stated by @celebdor we have test healthz at node and it's working fine

[core@rhhi-node-master-0 ~]$ curl -k https://0.0.0.0:6443/healthz
ok[core@rhhi-node-master-0 ~]$ 

[mcornea]

Looks like kube-apiserver container is not running, attaching the kube-apiserver container log:
kubeapiserver.log