SpringBoot Starter dependency and missing traces

[aswarcewicz]

Describe the bug

Not all traces have been logged in LoggingSpanExporter and not all traces have been forwarded to otel-collector.

Steps to reproduce

1. Use spring-cloud-gateway in version 4.1.1
2. add dependency to io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter and optionally to io.opentelemetry.instrumentation:opentelemetry-spring-webflux-5.3:2.0.0-alpha
3. put in application.yaml:

otel:
  logs:
    exporter: otlp,logging
  traces:
    sampler: always_on
    exporter: otlp,logging
  metrics:
    exporter: otlp,logging
  service:
    name: gateway
  exporter:
    otlp:
      enabled: true
      endpoint: http://localhost:4317
      protocol: grpc
    logging:
      enabled: true

4. run app and try some random requests

Expected behavior

all traces should appear in LoggingSpanExporter and all traces should be forwarded to otel-collector

Actual behavior

only some traces are appearing in LoggingSpanExporter

Javaagent or library instrumentation version

without agent, only spring dependencies

Environment

JDK:
OS:

Additional context

No response

[steverao]

Do you use the relevant capacity according to the docs following?

1. https://opentelemetry.io/docs/languages/java/automatic/spring-boot/
2. https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/spring/spring-webflux/spring-webflux-5.3/library

For the first question you mentioned above, I used same dependencies with you, I didn't reproduce your problem. Every traces was printed by LoggingSpanExporter.

For the second question, I found the traces can be exporter to collector correctly. The picture below is the traffic information collected by my own collector:

Maybe you can check whether miss some configurations according to above documents. Here is some dependencies information about otel I added based on docs.

 <dependencies>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-gateway</artifactId>
        <version>4.1.1</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry.instrumentation</groupId>
        <artifactId>opentelemetry-spring-boot-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry.instrumentation</groupId>
        <artifactId>opentelemetry-spring-webflux-5.3</artifactId>
    </dependency>
</dependencies>
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-bom</artifactId>
            <version>1.34.1</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
            <version>2.0.0-alpha</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

[aswarcewicz]

Thank you for your really fast response :)
Could you try this code: https://github.com/aswarcewicz/spring-demo-otel ?
After run app please enter localhost:8080 (possible multiple times to see logs from LoggingSpanExporter)

[laurit]

I couldn't see any of the traces being logged by the LoggingSpanExporter. In your sample app you are using both micrometer tracing and opentelemetry instrumentation. As far as I can tell the traces you get are from micrometer tracing, if you wish to use micrometer tracing then I'd suggest you ask assistance from the channels relevant to that project. If you wish to use opentelemetry instrumentation then firstly remove micrometer tracing, delete

    implementation("org.springframework.boot:spring-boot-starter-actuator")
    implementation("io.micrometer:micrometer-tracing")
    implementation("io.micrometer:micrometer-tracing-bridge-otel")

from build.gradle.kts and delete all the code that fails to compile after that. Now unfortunately currently io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter only enables spring webflux http client instrumentation, you have to enable the server manually (@zeitlinger @jeanbisutti should this also be done by the starter?). Add

    @Bean
    WebFilter telemetryFilter(OpenTelemetry openTelemetry) {
        return SpringWebfluxTelemetry.builder(openTelemetry)
                .build()
                .createWebFilterAndRegisterReactorHook();
    }

to a configuration class. Adjust the priority of your logging filter in https://github.com/aswarcewicz/spring-demo-otel/blob/e27ad1dfc25f656eef371fd0f43e6f243e67fa01/src/main/java/com/example/demootel/AccessLogging.java#L29 add + 10 there to ensure that tracing filter runs before it. For logback we provide 2 appenders, one is for exporting logs and the other one is for exposing trace and span id in mdc. You have already configured the first one but also need to configure the second one to see trace and span id in console output (@jeanbisutti @zeitlinger can starter configure this?). Check out the documentation in https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/logback/logback-mdc-1.0/library add implementation("io.opentelemetry.instrumentation:opentelemetry-logback-mdc-1.0") to build.gradle.kts and configure the appender in logback.xml. Add

    <appender name="otelConsole"
              class="io.opentelemetry.instrumentation.logback.mdc.v1_0.OpenTelemetryAppender">
        <appender-ref ref="console"/>
    </appender>

and replace usages of console with otelConsole. Also you have to modify the logging pattern, instead of %X{traceId},%X{spanId} use %X{trace_id},%X{span_id}. Now you should see the traces from otel instrumentation and also have trace id and span id in the console logs.

[zeitlinger]

webflux

yes, we should add this feature: #10415

[zeitlinger]

can starter configure log appender

yes, it will be automatically added with the next version #10306

[jeanbisutti]

can starter configure log appender

yes, it will be automatically added with the next version #10306

It's another OTel appender that will be automatically added. Issue created for the OTel Logaback MDC appender: #10416

[aswarcewicz]

I have switched to agent (maybe temporarily) because it provides environment data too. On the weekend I will try to follow your instructions @laurit Thanks!