Skip to content

Commit a39bb7f

Browse files
CharlieTLeCharlieTLe
committed
Allow for scoping TA to watch namespace
It's currently not possible to deploy the TA without cluster-wide permisisons. This change introduces a new env variable to the TA, WATCH_NAMESPACE, which allows for specifying which namespaces to watch. This approach is similar to how the opentelemetry-operator can be scoped to watch a single namespace. This does mean that cluster-wide resource like node metrics (cAdvisor) are no longer accessible, but this is acceptable since we only want the TA to know about targets that exist a specific namespaces. Fixes: #3086 Signed-off-by: Charlie Le <[email protected]>
1 parent d7291dd commit a39bb7f

File tree

9 files changed

+399
-10
lines changed

9 files changed

+399
-10
lines changed

‎.chloggen/namespace-ta.yaml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
2+
change_type: enhancement
3+
4+
# The name of the component, or a single word describing the area of concern, (e.g. collector, target allocator, auto-instrumentation, opamp, github action)
5+
component: target allocator
6+
7+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
8+
note: |
9+
Add support for `WATCH_NAMESPACE` environment variable in the target allocator.
10+
11+
# One or more tracking issues related to the change
12+
issues: [3086]
13+
14+
# (Optional) One or more lines of additional information to render under the primary note.
15+
# These lines will be padded with 2 spaces and then inserted directly into the document.
16+
# Use pipe (|) for multiline entries.
17+
subtext: |
18+
This variable can be set to an empty string to watch all namespaces, or to a comma-separated list of namespaces to watch.

‎cmd/otel-allocator/README.md

Lines changed: 84 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -191,9 +191,11 @@ Upstream documentation here: [PrometheusReceiver](https://github.com/open-teleme
191191

192192
### RBAC
193193

194-
Before the TargetAllocator can start scraping, you need to set up Kubernetes RBAC (role-based access controls) resources. This means that you need to have a `ServiceAccount` and corresponding cluster roles so that the TargetAllocator has access to all of the necessary resources to pull metrics from.
194+
Before the TargetAllocator can start scraping, you need to set up Kubernetes RBAC (role-based access controls) resources. This means that you need to have a `ServiceAccount` and corresponding ClusterRoles/Roles so that the TargetAllocator has access to all the necessary resources to pull metrics from.
195195

196-
You can create your own `ServiceAccount`, and reference it in `spec.targetAllocator.serviceAccount` in your `OpenTelemetryCollector` CR. You’ll then need to configure the `ClusterRole` and `ClusterRoleBinding` for this `ServiceAccount`, as per below.
196+
You can create your own `ServiceAccount`, and reference it in `spec.targetAllocator.serviceAccount` in your `OpenTelemetryCollector` CR. You’ll then need to configure the `ClusterRole` and `ClusterRoleBinding` or `Role` and `RoleBinding` for this `ServiceAccount`, as per below.
197+
198+
#### Cluster-scoped RBAC
197199

198200
```yaml
199201
targetAllocator:
@@ -204,11 +206,11 @@ You can create your own `ServiceAccount`, and reference it in `spec.targetAlloca
204206
```
205207

206208
> 🚨 **Note**: The Collector part of this same CR *also* has a serviceAccount key which only affects the collector and *not*
207-
the TargetAllocator.
209+
> the TargetAllocator.
208210

209-
If you omit the `ServiceAccount` name, the TargetAllocator creates a `ServiceAccount` for you. The `ServiceAccount`’s default name is a concatenation of the Collector name and the `-targetallocator` suffix. By default, this `ServiceAccount` has no defined policy, so you’ll need to create your own `ClusterRole` and `ClusterRoleBinding` for it, as per below.
211+
If you omit the `ServiceAccount` name, the TargetAllocator creates a `ServiceAccount` for you. The `ServiceAccount`’s default name is a concatenation of the Collector name and the `-targetallocator` suffix. By default, this `ServiceAccount` has no defined policy, so you’ll need to create your own `ClusterRole` and `ClusterRoleBinding` or `Role` and `RoleBinding` for it, as per below.
210212

211-
The role below will provide the minimum access required for the Target Allocator to query all the targets it needs based on any Prometheus configurations:
213+
The ClusterRole below will provide the minimum access required for the Target Allocator to query all the targets it needs based on any Prometheus configurations:
212214

213215
```yaml
214216
apiVersion: rbac.authorization.k8s.io/v1
@@ -242,7 +244,7 @@ rules:
242244
verbs: ["get"]
243245
```
244246

245-
If you enable the the `prometheusCR` (set `spec.targetAllocator.prometheusCR.enabled` to `true`) in the `OpenTelemetryCollector` CR, you will also need to define the following roles. These give the TargetAllocator access to the `PodMonitor` and `ServiceMonitor` CRs. It also gives namespace access to the `PodMonitor` and `ServiceMonitor`.
247+
If you enable the `prometheusCR` (set `spec.targetAllocator.prometheusCR.enabled` to `true`) in the `OpenTelemetryCollector` CR, you will also need to define the following ClusterRoles. These give the TargetAllocator access to the `PodMonitor` and `ServiceMonitor` CRs. It also gives namespace access to the `PodMonitor` and `ServiceMonitor`.
246248

247249
```yaml
248250
apiVersion: rbac.authorization.k8s.io/v1
@@ -263,8 +265,83 @@ rules:
263265
verbs: ["get", "list", "watch"]
264266
```
265267

266-
> ✨ The above roles can be combined into a single role.
268+
> ✨ The above ClusterRoles can be combined into a single ClusterRole.
269+
270+
#### Namespace-scoped RBAC
271+
272+
If you want to have the TargetAllocator watch a specific namespace, you can set the WATCH_NAMESPACE environment variable
273+
in the TargetAllocator's deployment. This is useful if you want to restrict the TargetAllocator to only watch Prometheus
274+
CRs in a specific namespace, and not have cluster-wide access.
275+
276+
```yaml
277+
targetAllocator:
278+
enabled: true
279+
serviceAccount: opentelemetry-targetallocator-sa
280+
prometheusCR:
281+
enabled: true
282+
env:
283+
- name: WATCH_NAMESPACE
284+
value: "foo"
285+
```
286+
287+
In this case, you will need to create a Role and RoleBinding instead of a ClusterRole and ClusterRoleBinding. The Role
288+
and RoleBinding should be created in the namespace specified in the WATCH_NAMESPACE environment variable.
267289

290+
```yaml
291+
apiVersion: rbac.authorization.k8s.io/v1
292+
kind: Role
293+
metadata:
294+
name: opentelemetry-targetallocator-role
295+
rules:
296+
- apiGroups:
297+
- ""
298+
resources:
299+
- pods
300+
- services
301+
- endpoints
302+
- configmaps
303+
- secrets
304+
- namespaces
305+
verbs:
306+
- get
307+
- watch
308+
- list
309+
- apiGroups:
310+
- apps
311+
resources:
312+
- statefulsets
313+
verbs:
314+
- get
315+
- watch
316+
- list
317+
- apiGroups:
318+
- discovery.k8s.io
319+
resources:
320+
- endpointslices
321+
verbs:
322+
- get
323+
- watch
324+
- list
325+
- apiGroups:
326+
- networking.k8s.io
327+
resources:
328+
- ingresses
329+
verbs:
330+
- get
331+
- watch
332+
- list
333+
- apiGroups:
334+
- monitoring.coreos.com
335+
resources:
336+
- servicemonitors
337+
- podmonitors
338+
- scrapeconfigs
339+
- probes
340+
verbs:
341+
- get
342+
- watch
343+
- list
344+
```
268345

269346
### Service / Pod monitor endpoint credentials
270347

@@ -420,4 +497,3 @@ Shards the received targets based on the discovered Collector instances
420497

421498
### Collector
422499
Client to watch for deployed Collector instances which will then provided to the Allocator.
423-

‎cmd/otel-allocator/internal/watcher/promOperator.go

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import (
88
"fmt"
99
"log/slog"
1010
"os"
11+
"strings"
1112
"time"
1213

1314
"github.com/blang/semver/v4"
@@ -53,7 +54,21 @@ func NewPrometheusCRWatcher(ctx context.Context, logger logr.Logger, cfg allocat
5354
return nil, err
5455
}
5556

56-
factory := informers.NewMonitoringInformerFactories(map[string]struct{}{v1.NamespaceAll: {}}, map[string]struct{}{}, mClient, allocatorconfig.DefaultResyncTime, nil) //TODO decide what strategy to use regarding namespaces
57+
// Check env var for WATCH_NAMESPACE and use it if its set, else use v1.NamespaceAll
58+
// This is to allow the operator to watch only a specific namespace
59+
watchNamespace, found := os.LookupEnv("WATCH_NAMESPACE")
60+
allowList := map[string]struct{}{}
61+
if found {
62+
logger.Info("watching namespace(s)", "namespaces", watchNamespace)
63+
for _, ns := range strings.Split(watchNamespace, ",") {
64+
allowList[ns] = struct{}{}
65+
}
66+
} else {
67+
allowList = map[string]struct{}{v1.NamespaceAll: {}}
68+
logger.Info("the env var WATCH_NAMESPACE isn't set, watching all namespaces")
69+
}
70+
71+
factory := informers.NewMonitoringInformerFactories(allowList, map[string]struct{}{}, mClient, allocatorconfig.DefaultResyncTime, nil) //TODO decide what strategy to use regarding namespaces
5772

5873
monitoringInformers, err := getInformers(factory)
5974
if err != nil {
@@ -99,7 +114,7 @@ func NewPrometheusCRWatcher(ctx context.Context, logger logr.Logger, cfg allocat
99114
logger.Error(err, "Retrying namespace informer creation in promOperator CRD watcher")
100115
return true
101116
}, func() error {
102-
nsMonInf, err = getNamespaceInformer(ctx, map[string]struct{}{v1.NamespaceAll: {}}, promLogger, clientset, operatorMetrics)
117+
nsMonInf, err = getNamespaceInformer(ctx, allowList, promLogger, clientset, operatorMetrics)
103118
return err
104119
})
105120
if getNamespaceInformerErr != nil {

‎tests/e2e-targetallocator/targetallocator-namespace/assert-jobs-succeeded.yaml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: batch/v1
2+
kind: Job
3+
metadata:
4+
name: check-metrics
5+
status:
6+
succeeded: 1
7+
---
8+
apiVersion: batch/v1
9+
kind: Job
10+
metadata:
11+
name: check-ta-jobs
12+
status:
13+
succeeded: 1
14+
---
15+
apiVersion: batch/v1
16+
kind: Job
17+
metadata:
18+
name: check-ta-scrape-configs
19+
status:
20+
succeeded: 1

‎tests/e2e-targetallocator/targetallocator-namespace/assert-workloads-ready.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
apiVersion: apps/v1
2+
kind: StatefulSet
3+
metadata:
4+
name: prometheus-cr-collector
5+
status:
6+
readyReplicas: 1
7+
replicas: 1
8+
---
9+
apiVersion: apps/v1
10+
kind: Deployment
11+
metadata:
12+
name: cr-targetallocator
13+
status:
14+
readyReplicas: 1
15+
replicas: 1

‎tests/e2e-targetallocator/targetallocator-namespace/chainsaw-test.yaml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json
2+
apiVersion: chainsaw.kyverno.io/v1alpha1
3+
kind: Test
4+
metadata:
5+
name: targetallocator-namespace
6+
spec:
7+
steps:
8+
- try:
9+
- apply:
10+
file: resources/rbac.yaml
11+
- apply:
12+
file: resources/otelcol.yaml
13+
- assert:
14+
file: assert-workloads-ready.yaml
15+
- apply:
16+
file: resources/jobs.yaml
17+
- assert:
18+
file: assert-jobs-succeeded.yaml
19+
catch:
20+
- podLogs:
21+
selector: app.kubernetes.io/managed-by=opentelemetry-operator

‎tests/e2e-targetallocator/targetallocator-namespace/resources/jobs.yaml

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
apiVersion: batch/v1
2+
kind: Job
3+
metadata:
4+
name: check-metrics
5+
spec:
6+
template:
7+
spec:
8+
restartPolicy: OnFailure
9+
containers:
10+
- name: check-metrics
11+
image: curlimages/curl
12+
args:
13+
- /bin/sh
14+
- -c
15+
- |
16+
for i in $(seq 30); do
17+
if curl -m 1 -s http://prometheus-cr-collector:9090/metrics | grep "otelcol"; then exit 0; fi
18+
sleep 5
19+
done
20+
exit 1
21+
---
22+
apiVersion: batch/v1
23+
kind: Job
24+
metadata:
25+
name: check-ta-jobs
26+
spec:
27+
template:
28+
spec:
29+
restartPolicy: OnFailure
30+
containers:
31+
- name: check-metrics
32+
image: curlimages/curl
33+
args:
34+
- /bin/sh
35+
- -c
36+
- curl -s http://cr-targetallocator/scrape_configs | grep "prometheus-cr"
37+
---
38+
apiVersion: batch/v1
39+
kind: Job
40+
metadata:
41+
name: check-ta-scrape-configs
42+
spec:
43+
template:
44+
spec:
45+
restartPolicy: OnFailure
46+
containers:
47+
- name: check-metrics
48+
image: curlimages/curl
49+
args:
50+
- /bin/sh
51+
- -c
52+
- curl -s http://cr-targetallocator/jobs | grep "prometheus-cr"

‎tests/e2e-targetallocator/targetallocator-namespace/resources/otelcol.yaml

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
apiVersion: opentelemetry.io/v1alpha1
2+
kind: TargetAllocator
3+
metadata:
4+
name: cr
5+
spec:
6+
args:
7+
"zap-log-level": "debug"
8+
prometheusCR:
9+
enabled: true
10+
scrapeInterval: 1s
11+
scrapeConfigSelector: {}
12+
probeSelector: {}
13+
serviceMonitorSelector: {}
14+
podMonitorSelector: {}
15+
observability:
16+
metrics:
17+
disablePrometheusAnnotations: true
18+
enableMetrics: true
19+
env:
20+
- name: WATCH_NAMESPACE
21+
value: "($namespace)"
22+
serviceAccount: ta
23+
---
24+
apiVersion: opentelemetry.io/v1beta1
25+
kind: OpenTelemetryCollector
26+
metadata:
27+
name: prometheus-cr
28+
labels:
29+
opentelemetry.io/target-allocator: cr
30+
spec:
31+
observability:
32+
metrics:
33+
disablePrometheusAnnotations: true
34+
enableMetrics: true
35+
config:
36+
receivers:
37+
prometheus:
38+
config:
39+
scrape_configs: []
40+
41+
processors:
42+
43+
exporters:
44+
prometheus:
45+
endpoint: 0.0.0.0:9090
46+
service:
47+
pipelines:
48+
metrics:
49+
receivers: [prometheus]
50+
exporters: [prometheus]
51+
telemetry:
52+
logs:
53+
level: "DEBUG"
54+
development: true
55+
encoding: "json"
56+
mode: statefulset
57+
serviceAccount: collector

0 commit comments

Comments
 (0)