How to pass jmx-metrics configuration parameters to the k8s opentelemetry auto-instrumentation injected java agent

[christian-sanchez-db]

Hi,

I hope I make myself clear to explain this question. I deployed in my k8s cluster the opentelemetry-operator and opentelemetry-collector using helm charts (https://opentelemetry.io/docs/kubernetes/getting-started/) with the objective to monitor a Java application, concretelly a JMX one.

To monitor this JMX application I have injected the java auto-instrumentation agent into the java k8s container using the open-telemetry operator and creating an "Instrumentation" Custom Resource like somehow is explained here https://opentelemetry.io/docs/kubernetes/operator/automatic/#java or here https://github.com/open-telemetry/opentelemetry-operator#opentelemetry-auto-instrumentation-injection

Now my next step should be to pass to this injected java auto-instrumentation agent the configuration to monitor the desired JMX MBeans like is required and explained here https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jmx-metrics/javaagent#configuration-files

However I couldn't figure out how to do it. I mean, in the previous doc is explained that for opentelemetry-java-instrumentation agent injected by a java command, this should be passed like

$ java -javaagent:path/to/opentelemetry-javaagent.jar \ -Dotel.jmx.config=path/to/config_file.yaml,more_rules.yaml \ ... \ -jar myapp.jar

but i don't understand how to relate this line -Dotel.jmx.config=path/to/config_file.yaml,more_rules.yaml to the java auto-instrumentation agent injected into my JMX application k8s container.

I searched in the API Reference for the opentelemetry Instrumentation CR (https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/api.md#instrumentation) but I can't see it there neither.

Could someone shed some light here please?

[laurit]

Instead of otel.jmx.config system property you could use environment variable OTEL_JMX_CONFIG if that helps.

[christian-sanchez-db]

Yes, I passed it as

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-autoinstrumentation
spec:
  exporter:
    endpoint: http://opentelemetry-collector.otel.svc:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1"
  java:
    image:  XXXXXXX/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.25.1
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://opentelemetry-collector.otel.svc:4317
      - name: OTEL_JMX_CONFIG
        # value: ./jmx_metrics_config.yaml 
        value: |
          ---
          rules:
            - bean: java.lang:type=Memory
              mapping:
                HeapMemoryUsage.used:
                  metric: my.own.jvm.heap.used
                  type: updowncounter
                  desc: The current heap size
                  unit: By
                HeapMemoryUsage.max:
                  metric: my.own.jvm.heap.max
                  type: updowncounter
                  desc: The maximum allowed heap size
                  unit: By
    resources:
      limits:
        memory: '400Mi'
        cpu: '200m'
      requests:
        memory: '200Mi'
        cpu: '100m' 

However I am not sure if passing directly the file value: ./jmx_metrics_config.yaml works, so I passed it as a yaml text block. I couldn't verify it because it seems that I am having other issues with otel-collector.

But I assume that this approach is valid.

[christian-sanchez-db]

Hi, I tried both options

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-autoinstrumentation
spec:
  exporter:
    endpoint: http://opentelemetry-collector.otel.svc:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1"
  java:
    image:  XXXXXX/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.25.1
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://opentelemetry-collector.otel.svc:4317
      - name: OTEL_JMX_CONFIG
        # value: ./jmx_metrics_config.yaml 
        value: |
          ---
          rules:
            - bean: java.lang:type=Threading
              mapping:
                ThreadCount:
                  metric: my.own.jvm.thread.count
                  type: updowncounter
                  desc: The current number of threads
                  unit: "1"
    resources:
      limits:
        memory: '400Mi'
        cpu: '200m'
      requests:
        memory: '200Mi'
        cpu: '100m'  

that's

   - name: OTEL_JMX_CONFIG
     value: ./jmx_metrics_config.yaml 

or

    - name: OTEL_JMX_CONFIG
      value: |
        ---
        rules:
          - bean: java.lang:type=Threading
            mapping:
              ThreadCount:
                metric: my.own.jvm.thread.count
                type: updowncounter
                desc: The current number of threads
                unit: "1"

but none works.

Anyone has tried to do something similar succesfully?

[christian-sanchez-db]

Also would be possible to have the java agent injected using the "Instrumentation" k8s CR as I did till now but passing the metric definitions via java command

$ java -Dotel.jmx.config=path/to/config_file.yaml,more_rules.yaml \
     ... \
     -jar myapp.jar

without -javaagent:path/to/opentelemetry-javaagent.jar ?

[christian-sanchez-db]

I could do like this. Injecting the javaagent using k8s Intrumentation CR and passing the otel.jmx.config files via java command -Dotel.jmx.config=path/to/config_file.yaml,more_rules.yaml

[oliver-zhang]

Where is your config_file.yaml? how to copy your config file to you app container?
I see the javaagent file is copyed to the container via opentelemetry operator

[christian-sanchez-db]

I created a ConfigMap with the yaml config files for all the containers that we want to monitor.

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-jmx-metrics
data:
  # file-like keys
  jwebconsole-jmx-metrics.yaml: |
    ---
    rules:
      - bean: java.lang:type=Threading
        mapping:
          ThreadCount:
            metric: dbcell.jwebconsole.jvm.thread.count
            type: updowncounter
            desc: The current number of threads
            unit: "1"
      - bean: java.lang:type=Memory
        mapping:
          HeapMemoryUsage.used:
            metric: dbcell.jwebconsole.jvm.heap.used
            type: updowncounter
            desc: The current heap size
            unit: By
          HeapMemoryUsage.max:
            metric: dbcell.jwebconsole.jvm.heap.max
            type: updowncounter
            desc: The maximum allowed heap size
            unit: By
  hive-account-jmx-metrics.yaml: |
    ---
    rules:
      - bean: java.lang:type=Threading
        mapping:
          ThreadCount:
            metric: dbcell.hiveaccount.jvm.thread.count
            type: updowncounter
            desc: The current number of threads
            unit: "1"
      - bean: java.lang:type=Memory
        mapping:
          HeapMemoryUsage.used:
            metric: dbcell.hiveaccount.jvm.heap.used
            type: updowncounter
            desc: The current heap size
            unit: By
          HeapMemoryUsage.max:
            metric: dbcell.hiveaccount.jvm.heap.max
            type: updowncounter
            desc: The maximum allowed heap size
            unit: By

then I mount this ConfigMap in all the pods where the containers live

volumes:
[...]
        - name: otel-config-volume
          configMap:
            name: otel-jmx-metrics
            items:
              - key: jwebconsole-jmx-metrics.yaml
                path: jwebconsole-jmx-metrics.yaml

containers:
          [...]
          volumeMounts:
          [...]
            - mountPath: /app/otelconfig/
              readOnly: true
              name: otel-config-volume

then I pass the location of the configfile to the java agent as -D java parameter

env:
             [...]
            - name: JAVA_TOOL_OPTIONS
              value: >-
                 [...]
                -Dotel.jmx.config=/app/otelconfig/jwebconsole-jmx-metrics.yaml
          args: # Docker image entrypoint is set to `java `

Finally the java agent, that is generated via Instrumentation CR and injected via opentelemetry operator into the container, get this JMX config file and can generate the desired metrics and send them to opentelemetry collector.

[oliver-zhang]

Yes, it is possible to do this, but it must modiy business yaml, this is unacceptable to us, so i want to modify the operator and copy the jmx-metrics config file to container.

[christian-sanchez-db]

I don't know how to do it in the way that you say.