Add liveness/readiness probes

[qjoly]

Is your feature request related to a problem? Please describe.
I'm having trouble with Minio, which can't cope with network failures of a few milliseconds (see #2458 #2450). I find myself with totally blocked and unusable operators. Restarting them by hand each time is out of the question. My tenants are no longer installed, nor are updates being triggered.

Describe the solution you'd like
I'd like to be able to put liveness (and readiness) probes on the pods. That way, I won't have to manually restart pods every time they get stuck.

Describe alternatives you've considered
I tried to look at the different endpoints I could use in a liveness probes, I had difficulties (it seems incompatible with the election of the leader). I'm sure you'll have some knowledge that I don't.

Thank you 😄

[ramondeklein]

Adding a liveness probe won't help anything. It will simply return that the operator is still running. To implement a proper readiness probe, we should know what to check for readiness. There are several things to check:

• API server (if STS is enabled)
• Resource synchronization thread

It looks like #2450 and #2458 affect resource watching, but we need to have a reproduction scenario to check this.

[qjoly]

It will simply return that the operator is still running

In this case, it's a design problem. A liveness that replies that “everything's fine” when the application is frozen is not a liveness.

---

I think there is something to do with the API exposed by operator's pods. We could use this API for the liveness/readiness probe. The challenge is that the API is only exposed on the leader.

This is just a suggestion, but maybe exposing the API on all pods (even if only the master uses it) would be a good start to add a liveness probe.

Currently :
On the working pod:

bash-5.1$ exec 3<>/dev/tcp/localhost/4221
bash-5.1$  # Exit code 0

On the stuck one:

bash-5.1$ exec 3<>/dev/tcp/localhost/4221
bash: connect: Connection refused
bash: /dev/tcp/localhost/4221: Connection refused
bash-5.1$

but we need to have a reproduction scenario to check this

I gave a reproduction scenario in #2450. All you need to do is cause a micro-network break between the workers and control-planes (or restart the API-Server).

If you need more details about my technical environment, please don't hesitate to ask. I'd be delighted to help you 🙂