Using Prometheus otel exporter with SpringBoot and Micrometer

[ziomill]

Hi guys,

I'm trying to set up my observability stack by using OpenTelemetry collector, Prometheus and Grafana. Our microservices are SpringBoot (2.6) based, shipped with Micrometer and instrumented with OpenTelemetry auto instrumentation lib.

Anyway, I'm experiencing that Prometheus metrics that are exposed directly by the actuator are quite different from Prometheus metrics that are exposed by the OpenTelemetry Collector.

For example:

These are the metrics exposed directly by the Actuator on the microservice endpoint http://localhost:10000/actuator/prometheus:
https://file.io/AJOXwYBd3JjQ

These are metrics exposed on the OpenTelemetry collector http://localhost:8889/metrics:
_https://file.io/o94eWqNZbJ3_t

This should be right because the OpenTelemetry instrumentation lib is going to translate Micrometer format into OpenTelemetry standard format.

Make it sense for you?

[trask]

hi @ziomill, I can't see the second link showing the collector metrics, can you check it?

[ziomill]

Hi @trask ,

You can download both files from here:
https://file.io/nmcZoH8fr4CD

Thank you

[Starefossen]

Sorry to chime on to an old discussion but I am having the exact same issue with this example application using the auto-instrumentation: spring-petclinic/spring-petclinic-kotlin@master...nais:spring-petclinic-kotlin:master

• localhost:8080/acturator/metrics: https://gist.github.com/Starefossen/1cdc17d105743927a2dd87c1365b3b31/raw/3ae92ebbeba670a890a3a5b2fd73ef076a7dd95d/micrometer-prometheus
• localhost:9464/metrics: https://gist.githubusercontent.com/Starefossen/1cdc17d105743927a2dd87c1365b3b31/raw/3ae92ebbeba670a890a3a5b2fd73ef076a7dd95d/otel-prometheus

[Starefossen]

@laurit I might have misunderstood something, must the application be run with the -javaagent or can the instrumentation be built completely into the resulting jar without the need of an -javaagent?

[laurit]

Javaagent offers automatic instrumentation, you just add the -javaagent to jvm arguments and the agent will instrument as much as it can. It is also possible to manually instrument your application, see https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md. In Standalone Library Instrumentation column we link to manual instrumentation libraries that you can include in your application and have to configure yourself to make the instrumentation work. You can write your own instrumentation code and use it together with either automatic instrumentation or other manual instrumentation libraries. So the answer to your question is that you don't have to use the javaagent, but it is easier to use it.

[laurit]

I tried https://github.com/nais/spring-petclinic-kotlin and for me it worked when I used -javaagent:.. -Dotel.metrics.exporter=prometheus, there were metrics that are from spring boot not otel in output. Guessing that by autoinstrumentation you didn't mean javaagent I also tried without it, for this I used prometheus exporter in the collector and it also seemed to work. When I removed https://github.com/nais/spring-petclinic-kotlin/blob/3cb3d3823476c829e4a1ca4c317fa0a07a3ed359/src/main/kotlin/org/springframework/samples/petclinic/PetClinicApplication.kt#L38 it didn't work and produced something similar that you had (I hope I remember correctly what I saw). I also tried removing https://github.com/nais/spring-petclinic-kotlin/blob/3cb3d3823476c829e4a1ca4c317fa0a07a3ed359/src/main/kotlin/org/springframework/samples/petclinic/config/OtelConfig.kt#L15 but that didn't seem to have an effect. One thing I noticed is that your app has a bit of a dependency mess going on https://github.com/nais/spring-petclinic-kotlin/blob/3cb3d3823476c829e4a1ca4c317fa0a07a3ed359/build.gradle.kts#L98 is not the correct dependency, it was renamed to "io.opentelemetry:opentelemetry-exporter-otlp a long time ago. Secondly io.spring.dependency-management also sets version for otel dependencies so you end with a mix of dependencies from different versions. Use ./gradlew -q dependencies --configuration runtimeClasspath to inspect what dependencies you really get. For me it got sorted with

dependencyManagement {
    imports {
        mavenBom("io.opentelemetry:opentelemetry-bom:$otelVersion")
    }
}

if needed also add io.opentelemetry:opentelemetry-bom-alpha, io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom and io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-apha. After that it worked. It seems that https://github.com/nais/spring-petclinic-kotlin/blob/3cb3d3823476c829e4a1ca4c317fa0a07a3ed359/src/main/kotlin/org/springframework/samples/petclinic/config/OtelConfig.kt#L15 is needed, adding it automatically does not appear to be working for this application.

[Starefossen]

Thank you so much for helping me, @laurit 🙏🏻 This is very appreciated!

I am beginning to see that the javaagent has some clear advantages in this scenario, but I am looking into all ways of instrumenting java applications as we will be starting our journey to instrument close to 2000 micro-services for the Norwegian Labor and Welfare Administration (@navikt) - about 80% of them written in Java.

There will probably have to be a good mix of different ways to instrument the applications as the teams and applications are quit different when it comes to how much legacy they have to dealt with and their technology preferences.

I have updated the application with your suggestions: nais/spring-petclinic-kotlin@3c06353 without seing much difference in the metrics produced by the otel prometheus exporter:

# TYPE target info
# HELP target Target metadata
target_info{service_name="petclinic",service_namespace="myteam",telemetry_sdk_language="java",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.30.1"} 1
# TYPE otel_scope_info info
# HELP otel_scope_info Scope metadata
otel_scope_info{otel_scope_name="io.opentelemetry.exporters.otlp-grpc"} 1
# TYPE otel_scope_info info
# HELP otel_scope_info Scope metadata
otel_scope_info{otel_scope_name="io.opentelemetry.sdk.trace"} 1
# TYPE otlp_exporter_seen_total counter
# HELP otlp_exporter_seen_total 
otlp_exporter_seen_total{otel_scope_name="io.opentelemetry.exporters.otlp-grpc",type="span"} 1.0 1696581724330
# TYPE otlp_exporter_exported_total counter
# HELP otlp_exporter_exported_total 
otlp_exporter_exported_total{otel_scope_name="io.opentelemetry.exporters.otlp-grpc",success="true",type="span"} 1.0 1696581724330
# TYPE queueSize_ratio gauge
# HELP queueSize_ratio The number of spans queued
queueSize_ratio{otel_scope_name="io.opentelemetry.sdk.trace",spanProcessorType="BatchSpanProcessor"} 0.0 1696581724330

[laurit]

Try with my fork laurit/spring-petclinic-kotlin@ace1b3f