Description

We find that the zookeeper operator, after deleting the PVC and removing the finalizer for a ZookeeperCluster CR with a non-zero DeletionTimestamp (in reconcileFinalizers), will continue running the reconciliation logic if there is no error:

func (r *ReconcileZookeeperCluster) reconcileFinalizers(instance *zookeeperv1beta1.ZookeeperCluster) (err error) {
	...
	if instance.DeletionTimestamp.IsZero() {
		...
	} else {
		if utils.ContainsString(instance.ObjectMeta.Finalizers, utils.ZkFinalizer) {
			if err = r.cleanUpAllPVCs(instance); err != nil {
				return err
			}
			instance.ObjectMeta.Finalizers = utils.RemoveString(instance.ObjectMeta.Finalizers, utils.ZkFinalizer)
			if err = r.client.Update(context.TODO(), instance); err != nil {
				return err
			}
		}
	}
	return nil
}

This can lead to unexpected resource creation, because after removing the finalizer of the CR, the CR is going to be deleted. When the CR is deleted, the resources (e.g., sts, service, secret, etc.) that are owned by the CR will also be deleted. If the operator does not immediately return after removing the finalizer, the original reconcile logic will find these resources do not exist and try to create these resources. For example, in reconcileStatefulSet the operator will create the zookeeper statefulset. Such creation is unexpected and unnecessary, as they will be deleted again since the owner CR does not exist.

Importance

should-have

Location

reconcileFinalizers:
https://github.com/srteam2020/zookeeper-operator/blob/a99c93ba06fbece2c262f7744517c2e537491c29/pkg/controller/zookeepercluster/zookeepercluster_controller.go#L664

Suggestions for an improvement

Instead of creating the resources that are destined to be deleted immediately, we should make the operator return from reconcile if the pvc is deleted successfully and the finalizer is removed from the CR successfully, as no further actions should be taken if the CR is deleted.

We are willing to send a PR to help fix it.