Additional Thread Metrics

[akats7]

Is your feature request related to a problem? Please describe.

The current scope of thread metrics appears to be limited to thread count, there are other thread based metrics that are rather critical, such as thread cpu time and metrics based on thread state.

Describe the solution you'd like

Add additional thread metrics for:

jvm.thread.cpu_time
jvm.thread.user_time

Describe alternatives you've considered

Using the JMX Gatherer

Additional context

No response

[steverao]

Could you clarify whether the metrics jvm.thread.blocked, jvm.thread.waiting and jvm.thread.timed_waiting represent the number of threads in the corresponding states?

[akats7]

Yeah that is what I meant, but now see that there's a count metric emitted per state, in that case just CPU and User time seems to be a gap

[trask]

hi @akats7!

what attributes would you propose on jvm.thread.cpu_time / jvm.thread.user_time?

[akats7]

Hey @trask,

I'd have to dig a bit into the internals of the runtime metric modules, but one approach could be to just support this for JMX

[akats7]

@trask can we add cpu time to runtime metrics through the thread MBean using ManagementFactory. Then we rely on mbean operations to get the time values. I get there may be cardinality concerns since thread name/pool name would have to be an attribute, so it can be disabled by default.

[trask]

@SylvainJuge @robsunday I'm hesitant for people to add new JMX metrics in the middle of your convergence effort, so would like to defer to you here

[akats7]

Thanks @trask! I do want to point out that these are rather important metrics. We've had a lot of internal requests for this from users who are migrating from vendor products that supported this out of the box.

[SylvainJuge]

To expand a bit on the "convergence effort" context, we are currently trying to add JVM metrics in a YAML descriptor with #13392, this YAML will NOT be directly used by instrumentation but will in the future be used by jmx-scraper which is a CLI program replacing JMX Gatherer, but using the same JMX implementation as instrumentation (and thus inheriting it's yaml support).

What we are currently focusing on for JVM metrics in YAML, is the ability to capture them in a way that is compliant with semantic conventions, which is already done by the instrumentation/runtime-telemetry modules but with code.

The instrumentation/runtime-telemetry uses code and JMX listeners that can't be replicated with YAML, thus some of the metrics we can capture with YAML can't be exactly replicated.
For example, with jvm.thread.count we can't capture the jvm.thread.state or jvm.thread.daemon. In short, depending on how the metric is captured we might or might not provide the expected details and attributes, which then makes those attributes as optional/recommended only.

I think we can add new metrics even if the current work is still in-progress, I would suggest to do that in a few steps:

• experiment with yaml to see if and how those could be captured remotely through JMX with jmx-scraper
• discuss their definition here
• add them to runtime-telemetry modules to validate we can capture them as expected
• contribute their definitions to semantic conventions as experimental (this could mean having to change the implementations done previously).
• add their semconv-compliant definitions to jvm.yaml that is being added in jmx add jvm metrics yaml #13392 , which hopefully would have been merged in the mean time.

As a temporary work-around, if you are able to capture those with YAML configuration, you should be able to provide a YAML file for them. However this is not a great OOTB experience, could easily break if the metric definition changes when adding it to semconv.

[akats7]

Hey @SylvainJuge, thanks for the context. So part of the issue is that I believe the jmx-scraper is only able to scrape attributes and not execute operations which would be required for these metrics.

In regards to the experimentation, I've already done this with the JMX Gatherer since it allows you to directly interact with the mbeans if using a custom script. However since the gatherer instruments also only allow the use of attributes, I had to rely on transformation closures to overwrite other mbeans which is not ideal.

Also, if possible part of this ask is to be able to move away from the remote approach, I might be missing something but is there a reason its preferable to interact with a JMX server vs just scraping it directly since the javaagent runs in the same JVM?

[SylvainJuge]

Also, if possible part of this ask is to be able to move away from the remote approach, I might be missing something but is there a reason its preferable to interact with a JMX server vs just scraping it directly since the javaagent runs in the same JVM?

Ideally, we should not force users to deploy an instrumentation agent to capture runtime metrics if those could be obtained externally with JMX scraper or gatherer.

However, we already have the case of some metrics that can't be captured without instrumentation and explicit code as they can only be captured from within the JVM, either because they require advanced JMX features or rely on JFR events. So this is something we can do already, but it adds more constraints on the users, for example the JVM metrics are not exactly the same if using Java 17 or Java 8, which could lead to user confusion or missed expectations.

If I understand it correctly, those metrics would be more in the "runtime-telemetry only" and would be very unlikely supported through YAML due to needing some post-processing, is that correct ?

Also, could you try to elaborate a bit on their definitions/attributes and from which MBean attribute would they be captured ?