User defined service monitors get deleted if name would match ones that supposed to be created by operator.

[maratsal]

Describe the bug
Not sure if this is a bug, but rather surprising behaviour.
User defined service monitors get deleted if name would match ones that supposed to be created by operator if deployment of
prometheus enabled. Basically operator checks if there are any currently defined service monitors in the namespace and if there are and their name is matching ones that operator would create, operator is deleting them. Please see part of the code on following link -

argocd-operator/controllers/argocd/prometheus.go

Lines 121 to 156 in 516e937

	// newServiceMonitorWithSuffix returns a new ServiceMonitor instance for the given ArgoCD using the given suffix.
	func newServiceMonitorWithSuffix(suffix string, cr *argoprojv1a1.ArgoCD) *monitoringv1.ServiceMonitor {
	return newServiceMonitorWithName(fmt.Sprintf("%s-%s", cr.Name, suffix), cr)
	}
	// reconcileMetricsServiceMonitor will ensure that the ServiceMonitor is present for the ArgoCD metrics Service.
	func (r *ReconcileArgoCD) reconcileMetricsServiceMonitor(cr *argoprojv1a1.ArgoCD) error {
	sm := newServiceMonitorWithSuffix(common.ArgoCDKeyMetrics, cr)
	if argoutil.IsObjectFound(r.Client, cr.Namespace, sm.Name, sm) {
	if !cr.Spec.Prometheus.Enabled {
	// ServiceMonitor exists but enabled flag has been set to false, delete the ServiceMonitor
	return r.Client.Delete(context.TODO(), sm)
	}
	return nil // ServiceMonitor found, do nothing
	}
	if !cr.Spec.Prometheus.Enabled {
	return nil // Prometheus not enabled, do nothing.
	}
	sm.Spec.Selector = metav1.LabelSelector{
	MatchLabels: map[string]string{
	common.ArgoCDKeyName: nameWithSuffix(common.ArgoCDKeyMetrics, cr),
	},
	}
	sm.Spec.Endpoints = []monitoringv1.Endpoint{
	{
	Port: common.ArgoCDKeyMetrics,
	},
	}
	if err := controllerutil.SetControllerReference(cr, sm, r.Scheme); err != nil {
	return err
	}
	return r.Client.Create(context.TODO(), sm)
	}

To Reproduce
Steps to reproduce the behavior:

1. Create service monitors in the namespace with following names:
   CLUSTERNAME-metrics
   CLUSTERNAME-server-metrics
   CLUSTERNAME-repo-server-metrics
2. Deploy argocd cluster with name CLUSTERNAME
3. operator will wipe created service monitors if prometheus deployment is not enabled for argocd cluster in its manifest.

Expected behavior
Service monitors would still exist.

Screenshots
N/A

Additional context
I would expect to check service monitor matching not only by name, but also by labels maybe? As a workaround of this problem I just change the names of service monitors and appended -custom suffix to it.

[Obirah]

Holy shit, thank you for this issue. I was going crazy over this. 🤡

[maratsal]

Holy shit, thank you for this issue. I was going crazy over this. 🤡

I am glad this helped you. I spent half a day to troubleshoot this. The best to fix this would be giving thumbs up to the original post 👍🏻 I guess :)