Skip to content

Files

Latest commit

author
Mateusz Rzeszutek
Rename metrics instrumentations (signalfx#769)
May 6, 2022
7cce6a0 · May 6, 2022

History

History
210 lines (157 loc) · 11.3 KB

metrics.md

File metadata and controls

210 lines (157 loc) · 11.3 KB

The official Splunk documentation for this page is Metrics and attributes of the Java agent. For instructions on how to contribute to the docs, see CONTRIBUTING.md.

Metrics

🚧  Status: Experimental - exported metric data and configuration properties may change.

The Splunk Distribution of OpenTelemetry Java agent gathers basic application metrics. Micrometer and Micrometer SignalFx registry gather and export metrics to either SignalFx SmartAgent or the Splunk distribution of OpenTelemetry Collector.

Because this feature is still experimental, metrics are not enabled by default. To enable metrics, add -Dsplunk.metrics.enabled=true to the JVM arguments or set the environment variable SPLUNK_METRICS_ENABLED to true. For more information, please see the advanced configuration for details.

Default metric tags

The following dimensions are automatically added to all metrics exported by the agent:

Tag name Tag value
deployment.environment The value of the deployment.environment resource attribute, if present.
runtime The value of the process.runtime.name resource attribute, e.g. OpenJDK Runtime Environment.
process.pid The Java process identifier (PID).
service The value of the service.name resource attribute.

Manual instrumentation

The Splunk Distribution of OpenTelemetry Java agent detects if the instrumented application is using Micrometer and injects a special MeterRegistry implementation that allows the agent to pick up custom user-defined meters, as long as they're registered in the global Metrics.globalRegistry instance provided by the Micrometer library.

Dependencies

You'll need to add a dependency on the micrometer-core library to be able to export custom metrics with the javaagent.

For Maven users:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-core</artifactId>
    <version>1.8.0</version>
</dependency>

For Gradle users:

implementation("io.micrometer:micrometer-core:1.8.0")

The agent supports all micrometer versions starting from 1.3.

Adding custom metrics

You can use one of meter factory methods provided by the Metrics class, or use meter builders and refer to the Metrics.globalRegistry directly:

class MyClass {
  Counter myCounter = Metrics.counter("my_custom_counter");
  Timer myTimer = Timer.builder("my_custom_timer").register(Metrics.globalRegistry);

  int foo() {
    myCounter.increment();
    return myTimer.record(this::fooImpl);
  }

  private int fooImpl() {
    // ...
  }
}

For more details on using the Micrometer API please consult the Micrometer docs.

Supported libraries

The following metrics are currently gathered by the agent:

Library/Framework Instrumentation name Versions
JVM metrics jvm-metrics-splunk Java runtimes version 8 and higher
Apache DBCP2 connection pool metrics commons-dbcp2-splunk 2.0 and higher
c3p0 connection pool metrics c3p0-splunk 0.9.5 and higher
HikariCP connection pool metrics hikaricp-splunk 3.0 and higher
Oracle Universal Connection Pool metrics oracle-ucp-splunk 11.2.0.4 and higher
Tomcat JDBC connection pool metrics tomcat-jdbc-splunk 8.5 and higher
Vibur DBCP connection pool metrics vibur-dbcp-splunk 20.0 and higher
Tomcat thread pool metrics tomcat 8.5 and higher
WebSphere Liberty web request thread pool liberty 20.0.0.12
WebLogic thread pools weblogic 12.x and 14.x

JVM

We use the built-in Micrometer JVM metrics extension to register JVM measurements.

Classloader metrics

Metric name Instrument Description
runtime.jvm.classes.loaded Gauge The number of loaded classes.
runtime.jvm.classes.unloaded Counter The total number of unloaded classes since the process started.

GC metrics

Metric name Instrument Description
runtime.jvm.gc.concurrent.phase.time Timer Time spent in concurrent phase, in milliseconds.
runtime.jvm.gc.live.data.size Gauge Size of long-lived heap memory pool after reclamation, in bytes.
runtime.jvm.gc.max.data.size Gauge Max size of long-lived heap memory pool, in bytes.
runtime.jvm.gc.memory.allocated Counter Incremented for an increase in the size of the young heap memory pool after one garbage collection to before the next.
runtime.jvm.gc.memory.promoted Counter Count of positive increases in the size of the old generation memory pool before garbage collection to after garbage collection.
runtime.jvm.gc.pause Timer Time spent in garbage collection pause, in milliseconds.

Memory metrics

Metric name Instrument Description
runtime.jvm.memory.committed Gauge The amount of memory guaranteed to be available for use to the Java virtual machine, in bytes.
runtime.jvm.memory.max Gauge The maximum amount of memory that can be used for memory management, in bytes.
runtime.jvm.memory.used Gauge The amount of used memory, in bytes.

All memory pool metrics have the following tags:

Tag name Tag value
area Either heap or nonheap.
id Name of the memory pool, e.g. Perm Gen.

Thread metrics

Metric name Instrument Description
runtime.jvm.threads.daemon Gauge The current number of live daemon threads.
runtime.jvm.threads.live Gauge The current number of live threads, including both daemon and non-daemon threads.
runtime.jvm.threads.peak Gauge The peak live thread count since the Java virtual machine started or the peak was reset.
runtime.jvm.threads.states Gauge The current number of threads per state (metric tag).

Connection pool metrics

Splunk Distribution of OpenTelemetry Java instruments several JDBC connection pool implementations:

Each of the supported connection pools reports a subset of the following metrics:

Metric name Instrument Description
db.pool.connections Gauge The number of open connections.
db.pool.connections.active Gauge The number of open connections that are currently in use.
db.pool.connections.idle Gauge The number of open connections that are currently idle.
db.pool.connections.idle.max Gauge The maximum number of idle open connections allowed.
db.pool.connections.idle.min Gauge The minimum number of idle open connections allowed.
db.pool.connections.max Gauge The maximum number of open connections allowed.
db.pool.connections.pending_threads Gauge The number of threads that are currently waiting for an open connection.
db.pool.connections.timeouts Counter The number of connection timeouts that have happened since the application start.
db.pool.connections.create_time Timer The time it took to create a new connection.
db.pool.connections.wait_time Timer The time it took to get an open connection from the pool.
db.pool.connections.use_time Timer The time between borrowing a connection and returning it to the pool.

All connection pool metrics have the following tags:

Tag name Tag value
pool.name The name of the connection pool: Spring bean name if Spring is used, the JMX object name otherwise.
pool.type The type/implementation of the connection pool: e.g. c3p0, dbcp2, hikari, oracle-ucp, tomcat-jdbc, vibur-dbcp.

Thread pool metrics

Splunk Distribution of OpenTelemetry Java instruments several thread pool implementations:

Each of the supported connection pools reports a subset of the following metrics:

Metric name Instrument Description
executor.threads Gauge The current number of threads in the pool.
executor.threads.active Gauge The number of threads that are currently busy.
executor.threads.idle Gauge The number of threads that are currently idle.
executor.threads.core Gauge Core thread pool size - the number of threads that are always kept in the pool.
executor.threads.max Gauge The maximum number of threads in the pool.
executor.tasks.submitted Counter The total number of tasks that were submitted to this executor.
executor.tasks.completed Counter The total number of tasks completed by this executor.

All thread pool metrics have the following tags:

Tag name Tag value
executor.name The name of the thread pool.
executor.type The type/implementation of the thread pool: e.g. tomcat, liberty, weblogic.