Missing apiserver metrics when using TA in per-node mode

[mcanevet]

Component(s)

target allocator

What happened?

Description

When using Target Allocator in per-node mode, it looks like I'm missing apiserver metrics.

Steps to Reproduce

I have this ServiceMonitor declaration (coming from kube-prometheus-stack helm chart):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-prometheus-stack-apiserver
spec:
  endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      metricRelabelings:
        - action: drop
          regex: >-
            apiserver_request_duration_seconds_bucket;(0.15|0.2|0.3|0.35|0.4|0.45|0.6|0.7|0.8|0.9|1.25|1.5|1.75|2|3|3.5|4|4.5|6|7|8|9|15|25|40|50)
          sourceLabels:
            - __name__
            - le
      port: https
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: false
        serverName: kubernetes
  jobLabel: component
  namespaceSelector:
    matchNames:
      - default
  selector:
    matchLabels:
      component: apiserver
      provider: kubernetes

When using Target Allocator in Deployment mode, I have some target in my apiserver job, by when using it in per-node mode, all my targets are empty:

curl http://localhost:8080/jobs/serviceMonitor%2Fmonitoring%2Fkube-prometheus-stack-apiserver%2F0/targets | jq

{
  "otel-collector-48c42": {
    "_link": "/jobs/serviceMonitor%2Fmonitoring%2Fkube-prometheus-stack-apiserver%2F0/targets?collector_id=otel-collector-48c42",
    "targets": []
  },
  "otel-collector-hsnf4": {
    "_link": "/jobs/serviceMonitor%2Fmonitoring%2Fkube-prometheus-stack-apiserver%2F0/targets?collector_id=otel-collector-hsnf4",
    "targets": []
  },
  "otel-collector-hkh9h": {
    "_link": "/jobs/serviceMonitor%2Fmonitoring%2Fkube-prometheus-stack-apiserver%2F0/targets?collector_id=otel-collector-hkh9h",
    "targets": []
  },
  "otel-collector-rzcrd": {
    "_link": "/jobs/serviceMonitor%2Fmonitoring%2Fkube-prometheus-stack-apiserver%2F0/targets?collector_id=otel-collector-rzcrd",
    "targets": []
  },
  "otel-collector-j82q6": {
    "_link": "/jobs/serviceMonitor%2Fmonitoring%2Fkube-prometheus-stack-apiserver%2F0/targets?collector_id=otel-collector-j82q6",
    "targets": []
  }
}

Expected Result

At least one collector scraps the apiserver targets.

Actual Result

No collector scraps the apiserver target resulting in no metrics.

Kubernetes Version

1.29.0

Operator version

0.96.0

Collector version

0.96.0

Environment information

No response

Log output

No response

Additional context

No response

[swiatekm]

At the moment, this is intended, though underdocumented. The per-node allocation strategy ignores targets without a Node. In #1828, there were several proposals for how this should work, but they weren't included in the original change to avoid additional complexity. Some also consider using a separate collector for control plane metrics to be the better choice, though this is of course debatable.

I'll update the documentation to make this limitation clear, and if you'd like, we can open a separate ticket to talk about changing it. Is that ok?

[mcanevet]

OK, thank you!