feat(ta): allowNamespaces and denyNamespaces

When the target allocator is configured to watch Prometheus custom
resources in the cluster to discover targets, it is currently hard-coded
to require a ClusterRole with a policy rule of listing namespaces. This
prevents usage in environments where configuring ClusterRoles is not
permitted i.e. in namespace-as-a-service setups where only Roles can be
created.

This change introduces two fields in the prometheusCR specification to
allow configuring the namespaces that can be interacted with by the
target allocator.

- allowNamespaces is a comma-separated list of namespaces for the target
  allocator to watch. If set to an empty string, it will list all
  list all namespaces in the cluster. This is mutually exclusive with
  denyNamespaces. Defaults to an empty string.
- denyNamespaces is a comma-separated list of namespaces for the target
  allocator to not watch. If set to an empty string, it will not exclude
  any namespaces. This is mutually exclusive with allowNamespaces.
  Defaults to an empty string.

Fixes: #3086

Signed-off-by: Charlie Le <[email protected]>

Author: CharlieTLe

.chloggen/namespace-ta.yaml

@@ -6,7 +6,7 @@ component: target allocator
 
 # A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
 note: |
-  Add support for `WATCH_NAMESPACE` environment variable in the target allocator.
+  Add support for setting the allowNamespaces and denyNamespaces in the target allocator.
 
 # One or more tracking issues related to the change
 issues: [3086]
@@ -15,4 +15,5 @@ issues: [3086]
 # These lines will be padded with 2 spaces and then inserted directly into the document.
 # Use pipe (|) for multiline entries.
 subtext: |
-  This variable can be set to an empty string to watch all namespaces, or to a comma-separated list of namespaces to watch.
+  allowNamespaces can be set to an empty string to watch all namespaces (default) or to a comma-separated list of namespaces to watch.
+  denyNamespaces can be set to an empty string to deny watching any namespaces (default) or to a comma-separated list of namespaces to deny watching.

apis/v1beta1/targetallocator_types.go

@@ -12,6 +12,12 @@ type TargetAllocatorPrometheusCR struct {
 	// Enabled indicates whether to use a PrometheusOperator custom resources as targets or not.
 	// +optional
 	Enabled bool `json:"enabled,omitempty"`
+	// AllowNamespaces Namespaces to scope the interaction of the Target Allocator and the apiserver (allow list). This is mutually exclusive with DenyNamespaces.
+	// +optional
+	AllowNamespaces string `json:"allowNamespaces,omitempty"`
+	// DenyNamespaces Namespaces to scope the interaction of the Target Allocator and the apiserver (allow list). This is mutually exclusive with AllowNamespaces.
+	// +optional
+	DenyNamespaces string `json:"denyNamespaces,omitempty"`
 	// Default interval between consecutive scrapes. Intervals set in ServiceMonitors and PodMonitors override it.
 	//Equivalent to the same setting on the Prometheus CR.
 	//

bundle/community/manifests/opentelemetry.io_opentelemetrycollectors.yaml

@@ -7889,6 +7889,10 @@ spec:
                     type: object
                   prometheusCR:
                     properties:
+                      allowNamespaces:
+                        type: string
+                      denyNamespaces:
+                        type: string
                       enabled:
                         type: boolean
                       podMonitorSelector:

bundle/community/manifests/opentelemetry.io_targetallocators.yaml

@@ -2257,6 +2257,10 @@ spec:
                 type: string
               prometheusCR:
                 properties:
+                  allowNamespaces:
+                    type: string
+                  denyNamespaces:
+                    type: string
                   enabled:
                     type: boolean
                   podMonitorSelector:

bundle/openshift/manifests/opentelemetry.io_opentelemetrycollectors.yaml

@@ -7889,6 +7889,10 @@ spec:
                     type: object
                   prometheusCR:
                     properties:
+                      allowNamespaces:
+                        type: string
+                      denyNamespaces:
+                        type: string
                       enabled:
                         type: boolean
                       podMonitorSelector:

bundle/openshift/manifests/opentelemetry.io_targetallocators.yaml

@@ -2257,6 +2257,10 @@ spec:
                 type: string
               prometheusCR:
                 properties:
+                  allowNamespaces:
+                    type: string
+                  denyNamespaces:
+                    type: string
                   enabled:
                     type: boolean
                   podMonitorSelector:

cmd/otel-allocator/README.md

@@ -269,8 +269,8 @@ rules:
 
 #### Namespace-scoped RBAC
 
-If you want to have the TargetAllocator watch a specific namespace, you can set the WATCH_NAMESPACE environment variable
-in the TargetAllocator's deployment. This is useful if you want to restrict the TargetAllocator to only watch Prometheus
+If you want to have the TargetAllocator watch a specific namespace, you can set the allowNamespaces field 
+in the TargetAllocator's prometheusCR configuration. This is useful if you want to restrict the TargetAllocator to only watch Prometheus
 CRs in a specific namespace, and not have cluster-wide access.
 
 ```yaml
@@ -279,13 +279,11 @@ CRs in a specific namespace, and not have cluster-wide access.
     serviceAccount: opentelemetry-targetallocator-sa
     prometheusCR:
       enabled: true
-    env:
-      - name: WATCH_NAMESPACE
-        value: "foo"
+      allowNamespaces: foo
 ```
 
 In this case, you will need to create a Role and RoleBinding instead of a ClusterRole and ClusterRoleBinding. The Role
-and RoleBinding should be created in the namespace specified in the WATCH_NAMESPACE environment variable.
+and RoleBinding should be created in the namespace specified by the allowNamespaces field.
 
 ```yaml
 apiVersion: rbac.authorization.k8s.io/v1

cmd/otel-allocator/internal/config/config.go

@@ -53,6 +53,8 @@ type Config struct {
 
 type PrometheusCRConfig struct {
 	Enabled                         bool                  `yaml:"enabled,omitempty"`
+	AllowNamespaces                 string                `yaml:"allow_namespaces,omitempty"`
+	DenyNamespaces                  string                `yaml:"deny_namespaces,omitempty"`
 	PodMonitorSelector              *metav1.LabelSelector `yaml:"pod_monitor_selector,omitempty"`
 	PodMonitorNamespaceSelector     *metav1.LabelSelector `yaml:"pod_monitor_namespace_selector,omitempty"`
 	ServiceMonitorSelector          *metav1.LabelSelector `yaml:"service_monitor_selector,omitempty"`

cmd/otel-allocator/internal/watcher/promOperator.go

@@ -54,21 +54,30 @@ func NewPrometheusCRWatcher(ctx context.Context, logger logr.Logger, cfg allocat
 		return nil, err
 	}
 
-	// Check env var for WATCH_NAMESPACE and use it if its set, else use v1.NamespaceAll
-	// This is to allow the operator to watch only a specific namespace
-	watchNamespace, found := os.LookupEnv("WATCH_NAMESPACE")
+	if cfg.PrometheusCR.AllowNamespaces != "" && cfg.PrometheusCR.DenyNamespaces != "" {
+		return nil, fmt.Errorf("both allow and deny namespaces are set, only one should be set")
+	}
+
 	allowList := map[string]struct{}{}
-	if found {
-		logger.Info("watching namespace(s)", "namespaces", watchNamespace)
-		for _, ns := range strings.Split(watchNamespace, ",") {
+	if cfg.PrometheusCR.AllowNamespaces != "" {
+		logger.Info("watching namespace(s)", "namespaces", cfg.PrometheusCR.AllowNamespaces)
+		for _, ns := range strings.Split(cfg.PrometheusCR.AllowNamespaces, ",") {
 			allowList[ns] = struct{}{}
 		}
 	} else {
+		logger.Info("cfg.PrometheusCR.AllowNamespaces is unset, watching all namespaces")
 		allowList = map[string]struct{}{v1.NamespaceAll: {}}
-		logger.Info("the env var WATCH_NAMESPACE isn't set, watching all namespaces")
 	}
 
-	factory := informers.NewMonitoringInformerFactories(allowList, map[string]struct{}{}, mClient, allocatorconfig.DefaultResyncTime, nil) //TODO decide what strategy to use regarding namespaces
+	denyList := map[string]struct{}{}
+	if cfg.PrometheusCR.DenyNamespaces != "" {
+		logger.Info("excluding namespace(s)", "namespaces", cfg.PrometheusCR.DenyNamespaces)
+		for _, ns := range strings.Split(cfg.PrometheusCR.DenyNamespaces, ",") {
+			denyList[ns] = struct{}{}
+		}
+	}
+
+	factory := informers.NewMonitoringInformerFactories(allowList, denyList, mClient, allocatorconfig.DefaultResyncTime, nil)
 
 	monitoringInformers, err := getInformers(factory)
 	if err != nil {
@@ -114,7 +123,7 @@ func NewPrometheusCRWatcher(ctx context.Context, logger logr.Logger, cfg allocat
 			logger.Error(err, "Retrying namespace informer creation in promOperator CRD watcher")
 			return true
 		}, func() error {
-			nsMonInf, err = getNamespaceInformer(ctx, allowList, promLogger, clientset, operatorMetrics)
+			nsMonInf, err = getNamespaceInformer(ctx, allowList, denyList, promLogger, clientset, operatorMetrics)
 			return err
 		})
 	if getNamespaceInformerErr != nil {
@@ -164,7 +173,7 @@ type PrometheusCRWatcher struct {
 	store                           *assets.StoreBuilder
 }
 
-func getNamespaceInformer(ctx context.Context, allowList map[string]struct{}, promOperatorLogger *slog.Logger, clientset kubernetes.Interface, operatorMetrics *operator.Metrics) (cache.SharedIndexInformer, error) {
+func getNamespaceInformer(ctx context.Context, allowList, denyList map[string]struct{}, promOperatorLogger *slog.Logger, clientset kubernetes.Interface, operatorMetrics *operator.Metrics) (cache.SharedIndexInformer, error) {
 	kubernetesVersion, err := clientset.Discovery().ServerVersion()
 	if err != nil {
 		return nil, err
@@ -180,7 +189,7 @@ func getNamespaceInformer(ctx context.Context, allowList map[string]struct{}, pr
 		clientset.CoreV1(),
 		clientset.AuthorizationV1().SelfSubjectAccessReviews(),
 		allowList,
-		map[string]struct{}{},
+		denyList,
 	)
 	if err != nil {
 		return nil, err

config/crd/bases/opentelemetry.io_opentelemetrycollectors.yaml

@@ -7875,6 +7875,10 @@ spec:
                     type: object
                   prometheusCR:
                     properties:
+                      allowNamespaces:
+                        type: string
+                      denyNamespaces:
+                        type: string
                       enabled:
                         type: boolean
                       podMonitorSelector:

config/crd/bases/opentelemetry.io_targetallocators.yaml

@@ -2254,6 +2254,10 @@ spec:
                 type: string
               prometheusCR:
                 properties:
+                  allowNamespaces:
+                    type: string
+                  denyNamespaces:
+                    type: string
                   enabled:
                     type: boolean
                   podMonitorSelector:

controllers/builder_test.go

@@ -1500,6 +1500,8 @@ config:
       target_label: instance
 filter_strategy: relabel-config
 prometheus_cr:
+  allow_namespaces: ""
+  deny_namespaces: ""
   enabled: true
   pod_monitor_selector: null
   probe_selector: null
@@ -1537,7 +1539,7 @@ prometheus_cr:
 									"app.kubernetes.io/version":    "latest",
 								},
 								Annotations: map[string]string{
-									"opentelemetry-targetallocator-config/hash": "286a5a4e7ec6d2ce652a4ce23e135c10053b4c87fd080242daa5bf21dcd5a337",
+									"opentelemetry-targetallocator-config/hash": "780df8ce12aa86e4c0c159a7550b74f12b0bede90a64b448af05b35205291f01",
 								},
 							},
 							Spec: corev1.PodSpec{
@@ -1669,7 +1671,7 @@ prometheus_cr:
 							"app.kubernetes.io/version":    "latest",
 						},
 						Annotations: map[string]string{
-							"opentelemetry-targetallocator-config/hash": "286a5a4e7ec6d2ce652a4ce23e135c10053b4c87fd080242daa5bf21dcd5a337",
+							"opentelemetry-targetallocator-config/hash": "780df8ce12aa86e4c0c159a7550b74f12b0bede90a64b448af05b35205291f01",
 						},
 					},
 					Spec: policyV1.PodDisruptionBudgetSpec{
@@ -1959,6 +1961,8 @@ config:
       target_label: instance
 filter_strategy: relabel-config
 prometheus_cr:
+  allow_namespaces: ""
+  deny_namespaces: ""
   enabled: true
   pod_monitor_selector: null
   probe_selector: null
@@ -1996,7 +2000,7 @@ prometheus_cr:
 									"app.kubernetes.io/version":    "latest",
 								},
 								Annotations: map[string]string{
-									"opentelemetry-targetallocator-config/hash": "286a5a4e7ec6d2ce652a4ce23e135c10053b4c87fd080242daa5bf21dcd5a337",
+									"opentelemetry-targetallocator-config/hash": "780df8ce12aa86e4c0c159a7550b74f12b0bede90a64b448af05b35205291f01",
 								},
 							},
 							Spec: corev1.PodSpec{
@@ -2128,7 +2132,7 @@ prometheus_cr:
 							"app.kubernetes.io/version":    "latest",
 						},
 						Annotations: map[string]string{
-							"opentelemetry-targetallocator-config/hash": "286a5a4e7ec6d2ce652a4ce23e135c10053b4c87fd080242daa5bf21dcd5a337",
+							"opentelemetry-targetallocator-config/hash": "780df8ce12aa86e4c0c159a7550b74f12b0bede90a64b448af05b35205291f01",
 						},
 					},
 					Spec: policyV1.PodDisruptionBudgetSpec{
@@ -2463,6 +2467,8 @@ https:
   tls_cert_file_path: /tls/tls.crt
   tls_key_file_path: /tls/tls.key
 prometheus_cr:
+  allow_namespaces: ""
+  deny_namespaces: ""
   enabled: true
   pod_monitor_selector: null
   probe_selector: null
@@ -2500,7 +2506,7 @@ prometheus_cr:
 									"app.kubernetes.io/version":    "latest",
 								},
 								Annotations: map[string]string{
-									"opentelemetry-targetallocator-config/hash": "3e2818ab54d866289de7837779e86e9c95803c43c0c4b58b25123e809ae9b771",
+									"opentelemetry-targetallocator-config/hash": "fbe8a4926da66590ccd42266083a98c88ac1f4970e1c662afc4a5841deca023f",
 								},
 							},
 							Spec: corev1.PodSpec{
@@ -2658,7 +2664,7 @@ prometheus_cr:
 							"app.kubernetes.io/version":    "latest",
 						},
 						Annotations: map[string]string{
-							"opentelemetry-targetallocator-config/hash": "3e2818ab54d866289de7837779e86e9c95803c43c0c4b58b25123e809ae9b771",
+							"opentelemetry-targetallocator-config/hash": "fbe8a4926da66590ccd42266083a98c88ac1f4970e1c662afc4a5841deca023f",
 						},
 					},
 					Spec: policyV1.PodDisruptionBudgetSpec{
@@ -3532,6 +3538,8 @@ config:
       target_label: instance
 filter_strategy: relabel-config
 prometheus_cr:
+  allow_namespaces: ""
+  deny_namespaces: ""
   enabled: true
   pod_monitor_selector: null
   probe_selector: null
@@ -3569,7 +3577,7 @@ prometheus_cr:
 									"app.kubernetes.io/version":    "latest",
 								},
 								Annotations: map[string]string{
-									"opentelemetry-targetallocator-config/hash": "f80c054419fe2f9030368557da143e200c70772d1d5f1be50ed55ae960b4b17d",
+									"opentelemetry-targetallocator-config/hash": "d03c739e0ad848160f39e963b358fa11c33abe4bb28be11e496a539314bc4b82",
 								},
 							},
 							Spec: corev1.PodSpec{
@@ -3701,7 +3709,7 @@ prometheus_cr:
 							"app.kubernetes.io/version":    "latest",
 						},
 						Annotations: map[string]string{
-							"opentelemetry-targetallocator-config/hash": "f80c054419fe2f9030368557da143e200c70772d1d5f1be50ed55ae960b4b17d",
+							"opentelemetry-targetallocator-config/hash": "d03c739e0ad848160f39e963b358fa11c33abe4bb28be11e496a539314bc4b82",
 						},
 					},
 					Spec: policyV1.PodDisruptionBudgetSpec{
@@ -3807,6 +3815,8 @@ config:
       target_label: instance
 filter_strategy: relabel-config
 prometheus_cr:
+  allow_namespaces: ""
+  deny_namespaces: ""
   enabled: true
   pod_monitor_selector: null
   probe_selector: null
@@ -3844,7 +3854,7 @@ prometheus_cr:
 									"app.kubernetes.io/version":    "latest",
 								},
 								Annotations: map[string]string{
-									"opentelemetry-targetallocator-config/hash": "f80c054419fe2f9030368557da143e200c70772d1d5f1be50ed55ae960b4b17d",
+									"opentelemetry-targetallocator-config/hash": "d03c739e0ad848160f39e963b358fa11c33abe4bb28be11e496a539314bc4b82",
 								},
 							},
 							Spec: corev1.PodSpec{
@@ -3976,7 +3986,7 @@ prometheus_cr:
 							"app.kubernetes.io/version":    "latest",
 						},
 						Annotations: map[string]string{
-							"opentelemetry-targetallocator-config/hash": "f80c054419fe2f9030368557da143e200c70772d1d5f1be50ed55ae960b4b17d",
+							"opentelemetry-targetallocator-config/hash": "d03c739e0ad848160f39e963b358fa11c33abe4bb28be11e496a539314bc4b82",
 						},
 					},
 					Spec: policyV1.PodDisruptionBudgetSpec{
@@ -4132,6 +4142,8 @@ config:
       target_label: instance
 filter_strategy: relabel-config
 prometheus_cr:
+  allow_namespaces: ""
+  deny_namespaces: ""
   enabled: true
   pod_monitor_selector: null
   probe_selector: null
@@ -4169,7 +4181,7 @@ prometheus_cr:
 									"app.kubernetes.io/version":    "latest",
 								},
 								Annotations: map[string]string{
-									"opentelemetry-targetallocator-config/hash": "286a5a4e7ec6d2ce652a4ce23e135c10053b4c87fd080242daa5bf21dcd5a337",
+									"opentelemetry-targetallocator-config/hash": "780df8ce12aa86e4c0c159a7550b74f12b0bede90a64b448af05b35205291f01",
 								},
 							},
 							Spec: corev1.PodSpec{
@@ -4301,7 +4313,7 @@ prometheus_cr:
 							"app.kubernetes.io/version":    "latest",
 						},
 						Annotations: map[string]string{
-							"opentelemetry-targetallocator-config/hash": "286a5a4e7ec6d2ce652a4ce23e135c10053b4c87fd080242daa5bf21dcd5a337",
+							"opentelemetry-targetallocator-config/hash": "780df8ce12aa86e4c0c159a7550b74f12b0bede90a64b448af05b35205291f01",
 						},
 					},
 					Spec: policyV1.PodDisruptionBudgetSpec{
@@ -4431,6 +4443,8 @@ https:
   tls_cert_file_path: /tls/tls.crt
   tls_key_file_path: /tls/tls.key
 prometheus_cr:
+  allow_namespaces: ""
+  deny_namespaces: ""
   enabled: true
   pod_monitor_selector: null
   probe_selector: null
@@ -4468,7 +4482,7 @@ prometheus_cr:
 									"app.kubernetes.io/version":    "latest",
 								},
 								Annotations: map[string]string{
-									"opentelemetry-targetallocator-config/hash": "3e2818ab54d866289de7837779e86e9c95803c43c0c4b58b25123e809ae9b771",
+									"opentelemetry-targetallocator-config/hash": "fbe8a4926da66590ccd42266083a98c88ac1f4970e1c662afc4a5841deca023f",
 								},
 							},
 							Spec: corev1.PodSpec{
@@ -4626,7 +4640,7 @@ prometheus_cr:
 							"app.kubernetes.io/version":    "latest",
 						},
 						Annotations: map[string]string{
-							"opentelemetry-targetallocator-config/hash": "3e2818ab54d866289de7837779e86e9c95803c43c0c4b58b25123e809ae9b771",
+							"opentelemetry-targetallocator-config/hash": "fbe8a4926da66590ccd42266083a98c88ac1f4970e1c662afc4a5841deca023f",
 						},
 					},
 					Spec: policyV1.PodDisruptionBudgetSpec{