How can I track down the source of metrics

[marcsaegesser]

I'm trying to track down where some metrics come from in Java applications that have been automatically instrumented using the agent.

I have a lot of Kubernetes workloads that use Tomcat (lots of different versions). What I'm seeing is that metrics that I expect to see, based on documentation and code, are not showing up (e.g. http.server.tomcat.*). But some workloads are publishing metrics that I don't see in the code or documentation (e.g. tomcat.connections.current). But, these metrics only appear for some workloads and not others. I have evidence that one workload was publishing these Tomcat metrics in the past but then stopped, and I can't find any code change that would appear to be related to that.

So, I'm looking for any pointers that people can provide to help me figure out where I should look to track down what metrics I should expect to see from Tomcat and why I might be getting some that I don't expect (but want) and not getting some that I do expect.

[laurit]

http.server.tomcat.* metrics are provided by https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jmx-metrics/javaagent#predefined-metrics you need to manually enable these. tomcat.connections.current is from micrometer https://github.com/micrometer-metrics/micrometer/blob/f9c939efcdce6e578d9d36c5d5fa0a258807e43f/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/tomcat/TomcatMetrics.java#L171

[marcsaegesser]

Thank you. I completely missed micrometer.