Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spring Webflux Context issue with TRACE_CONTEXT_KEY #13475

Closed
dlefler-chwy opened this issue Mar 7, 2025 · 8 comments
Closed

Spring Webflux Context issue with TRACE_CONTEXT_KEY #13475

dlefler-chwy opened this issue Mar 7, 2025 · 8 comments

Comments

@dlefler-chwy
Copy link

I have been attempting to get open telemetry working in my Spring Webflux application. The application uses a data dog java agent. Below is the configuration:

Spring boot version: 3.4.1
Spring webflux version: 6.2.1
opentelemetry-sdk version: 1.47.0
opentelemetry-api version: 1.47.0
opentelemetry-sdk-extension-autoconfigure: 1.47.0
opentelemetry-spring-webflux-5.3: 2.12.0-alpha

I am able to successfully instrument my application using just the open telemetry api. However, I want to add custom span attributes to my spans and in order to do this, I added opentelemetry-spring-webflux-5.3: 2.12.0-alpha. The reason I thought that this library would help access the correct span was that TRACE_CONTEXT_KEY would be added to the reactor context. When debugging my application, I do not see this in the reactor context and stepping through the code,storeOpenTelemetryContext is never called.

Is there something wrong with the way I have set this up? I have set up the correct beans following the README for the webflux library. Any help would be greatly appreciated!

@trask
Copy link
Member

trask commented Mar 10, 2025

The application uses a data dog java agent

hi @dlefler-chwy! I suspect you'll need to go to data dog for this question

@dlefler-chwy
Copy link
Author

The application uses a data dog java agent

hi @dlefler-chwy! I suspect you'll need to go to data dog for this question

Hi @trask! Thanks for the pointer. I took some time yesterday to try to see if I could get my code to run locally with the opentelemetry-javaagent as well. I was able to do this and experienced the same behavior.

I am relatively new to webflux as well as setting up observability but what would make you think that the opentelemetry-spring-webflux-5.3 library would not be working correctly due to the datadog agent? Is it something to do with the agent triggering a write to the reactor context for TRACE_CONTEXT_KEY or something like that? I see plenty of attempts to read TRACE_CONTEXT_KEY from the reactor context.

@trask
Copy link
Member

trask commented Mar 11, 2025

if you can create a minimal repro (preferably as a github repository with instructions) of the issue using the opentelemetry-javaagent that would be great

@dlefler-chwy
Copy link
Author

if you can create a minimal repro (preferably as a github repository with instructions) of the issue using the opentelemetry-javaagent that would be great

@trask here is a project reproing the issue: https://github.com/dlefler-chwy/webflux-otel-context-issue

@laurit
Copy link
Contributor

laurit commented Mar 14, 2025

I think the issue is that you are using global open telemetry in https://github.com/dlefler-chwy/webflux-otel-context-issue/blob/fe913a0547e203a53299d4c7fc291097ad288730/src/main/java/com/example/reactivewebservice/ReactiveWebServiceApplication.java#L22 Global open telemetry is meant to be used with the java agent where it returns the open telemetry instance configured by the agent. By default it will return a no-op instance that doesn't do anything. If you'd replace it with AutoConfiguredOpenTelemetrySdk.initialize() then your app would probably produce some spans. Secondly if you wish to add an attribute to current span you should use Span.current().setAttribute("foo", "bar"). Open telemetry context is not always set to reactor context.

@laurit laurit added the needs author feedback Waiting for additional feedback from the author label Mar 14, 2025
@dlefler-chwy
Copy link
Author

I think the issue is that you are using global open telemetry in https://github.com/dlefler-chwy/webflux-otel-context-issue/blob/fe913a0547e203a53299d4c7fc291097ad288730/src/main/java/com/example/reactivewebservice/ReactiveWebServiceApplication.java#L22 Global open telemetry is meant to be used with the java agent where it returns the open telemetry instance configured by the agent. By default it will return a no-op instance that doesn't do anything. If you'd replace it with AutoConfiguredOpenTelemetrySdk.initialize() then your app would probably produce some spans. Secondly if you wish to add an attribute to current span you should use Span.current().setAttribute("foo", "bar"). Open telemetry context is not always set to reactor context.

Thanks @laurit.

I tried running a debugger on the application start up for my sample application and the OpenTelemetry bean does not appear to be configuring to the no-op instance. I also attempted to use your suggested AutoConfiguredOpenTelemetrySdk which works but also results in the warning message:

io.opentelemetry.api.GlobalOpenTelemetry - You are currently using the OpenTelemetry Instrumentation Java Agent; all GlobalOpenTelemetry.set calls are ignored - the agent provides the global OpenTelemetry object used by your application

I think the warning makes sense since the sample project does use the agent so we would expect it to configure the OpenTelemty instance over the sdk.

As per the suggestion about adding to the span, shouldn't using Span.current().setAttribute("foo", "bar") not work in a reactive application? From what I can tell, this uses ThreadLocal to store the Span context which means that calling Span.current may return an incorrect Span for the current request. Is there something here I am missing?

@github-actions github-actions bot removed the needs author feedback Waiting for additional feedback from the author label Mar 17, 2025
@laurit
Copy link
Contributor

laurit commented Mar 17, 2025

I tried running a debugger on the application start up for my sample application and the OpenTelemetry bean does not appear to be configuring to the no-op instance. I also attempted to use your suggested AutoConfiguredOpenTelemetrySdk which works but also results in the warning message:
io.opentelemetry.api.GlobalOpenTelemetry - You are currently using the OpenTelemetry Instrumentation Java Agent; all GlobalOpenTelemetry.set calls are ignored - the agent provides the global OpenTelemetry object used by your application
I think the warning makes sense since the sample project does use the agent so we would expect it to configure the OpenTelemty instance over the sdk.

So you are using the agent. In that case GlobalOpenTelemetry.get() will return the OpenTelemetry instance configured by the agent. Since agent already includes the spring webflux instrumentation adding the same instrumentation again manually does not make sense.

As per the suggestion about adding to the span, shouldn't using Span.current().setAttribute("foo", "bar") not work in a reactive application? From what I can tell, this uses ThreadLocal to store the Span context which means that calling Span.current may return an incorrect Span for the current request. Is there something here I am missing?

Instrumentation restores the ThreadLocal context for the reactive callbacks so Span.current() should work.

@dlefler-chwy
Copy link
Author

Thanks @laurit for clarifying usage. I think that this thread can be closed as I think auto instrumentation should be able to support our use cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants