Exporting logs with MDC

[ofir-popowski]

I have a Spring Boot app and I've tried exporting logs using the java agent. This worked fine with the following JVM flags:

-javaagent:/opt/java-agent/opentelemetry-javaagent.jar
-Dotel.service.name=<my service>
-Dotel.logs.exporter=otlp
-Dotel.metrics.exporter=none
-Dotel.traces.exporter=none
-Dotel.exporter.otlp.headers=Authorization=Bearer <my token>
-Dotel.exporter.otlp.endpoint=<my endpoint>

This has worked fine, but anything I change in logback.xml has no effect on the structure of the logs being exported, specifically any MDC fields I add (the [%mdc] part in the pattern)

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger - [%mdc]: %msg%n
            </pattern>
        </encoder>
    </appender>
    <root level="INFO" additivity="false">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>

I also tried adding these JVM flags, and they did nothing

-Dotel.instrumentation.logback-appender.experimental.capture-logger-context-attributes=true
-Dotel.javaagent.logging=application

Since that did not work, I tried switching to the Spring Boot starter instead and followed the getting started, added the following properties to application.properties:

otel.exporter.otlp.logs.endpoint=<my endpoint>
otel.exporter.otlp.headers.Authorization=Bearer <my token>
otel.traces-exporter=none
otel.metrics-exporter=none
otel.logs-exporter=otlp

This just failed on Failed to bind properties under 'otel' to io.opentelemetry.instrumentation.spring.autoconfigure.internal.properties.OtelSpringProperties:

So what am I missing? Either of these options would be okay with me, I just want my logback.xml format and MDC fields exported

[laurit]

Try otel.instrumentation.logback-appender.experimental.capture-mdc-attributes, see https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/logback/logback-appender-1.0/javaagent/README.md

[laurit]

how exactly did you add it and what did you use as the value? Did you test it with the agent or spring starter?

[ofir-popowski]

Testing with the agent
I've added both of these to the JVM flags

  "-Dotel.instrumentation.logback-appender.experimental.capture-mdc-attributes=*",
  "-Dotel.instrumentation.logback-appender.experimental.capture-logger-context-attributes=true",

I also enabled debug logs for the agent, and I think the MDC map is there
2025-01-23 09:04:56.930 [main] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - [, env=local,tenant=tenant,user=user]

But I don't see it on the other end receiving those fields, so I'm starting to suspect that part might not get parsed correctly. I'll check it out and mark your comment as answer if my suspicion is correct

[laurit]

Mdc elements are added as attributes on the log record. You should be able verify that they are present by sending data to a collector that uses debug exporter.

[ofir-popowski]

Sorry for the late reply

I tried to add the attributes you've said, then I pointed the endpoint towards https://webhook.site, just to see the contents of everything that gets there
I still don't see anything from my MDC there :<

Am I misunderstanding the relationship between logback.xml and the opentelemetry agent?
Are those fields attached to the tracing part maybe?

My opentelemetry setup still looks like this:

  "-Dotel.javaagent.logging=application",
  "-Dotel.instrumentation.logback-appender.experimental.capture-mdc-attributes=*",
  "-Dotel.instrumentation.logback-appender.experimental.capture-logger-context-attributes=true",
  "-Dotel.service.name=my-service",
  "-Dotel.logs.exporter=otlp",
  "-Dotel.metrics.exporter=none",
  "-Dotel.traces.exporter=none",

[laurit]

If you wish for someone to take a look at this it would be best if you could provide a minimal application that reproduces your issues along with any necessary instructions. That way we can see whether this is a bug or something else.

[renan-ikigaidigital]

The docs in the repo suggests to use another logappender:

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/logback/logback-mdc-1.0/library/README.md

        <appender name="OTEL"
            class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
            <captureExperimentalAttributes>true</captureExperimentalAttributes>
            <captureKeyValuePairAttributes>true</captureKeyValuePairAttributes>
            <captureCodeAttributes>true</captureCodeAttributes>
            <captureMarkerAttribute>true</captureMarkerAttribute>
            <captureMdcAttributes>*</captureMdcAttributes>
        </appender>
        <root level="INFO">
            <appender-ref ref="STDOUT" />
            <appender-ref ref="OTEL" />
        </root>