Cluster status.phase gets "Failed" forever once FailureMessage or FailureReason is set

[fariaass]

What steps did you take and what happened?

There was a problem related to infrastructure, the provider reported the error to CAPI which wrote it to status.failureMessage and status.failureReason field in Cluster CR. The problem got resolved but the cluster phase was never updated and the error message is still there

What did you expect to happen?

I expected the status.phase to become "Provisioned" and the fields status.failureMessage and status.failureReason to be cleaned

Cluster API version

1.7.1

Kubernetes version

1.28.5

Anything else you would like to add?

I was reading CAPI code when I noticed that the fields status.failureMessage and status.failureReason are never updated if they aren't defined. So if they were set with an error, they would always have that error until they get a new error. The code where the fields are update (or not): https://github.com/kubernetes-sigs/cluster-api/blob/main/internal/controllers/cluster/cluster_controller_phases.go#L130-L144

Label(s) to be applied

/kind bug

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPI contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[adilGhaffarDev]

status.failureMessage and status.failureReason should only be set when failure is terminal, in other cases where the failure is recoverable you should not set status.failureMessage and status.failureReason

https://cluster-api.sigs.k8s.io/developer/providers/machine-infrastructure#behavior

[fariaass]

Got it! tks! Just a question, you sent the machine docs, is it the same for clusters?

[fabriziopandini]

yes it is the same for cluster

/kind support
/close

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

yes it is the same for cluster

/kind support
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[fabriziopandini]

fyi, in #10897 I'm proposing to get rid of the confusing concept of terminal failures