Store baggage so it's available to all future spans of same trace

[kbouck]

(warning: new to otel, so apologies if I use terms/concepts incorrectly or without precision)

I'm working on an custom extension to the java auto instrumentation for instrumenting an existing application (so, can't change source code, only instrumentation). I want the ability for baggage entries that are added anywhere during a trace to be available to add as span attributes of any future spans belonging to same trace, regardless of parent-child-ancestor relationship. Correct me if I am wrong, but it seems that this only works if the baggage was added in a scope that is current or an ancestor.

Consider this trace (comments indicating what I wish to do):

• Root Span A
  • Span B
    • Span B1 // Set Baggage Entry in an AttributesExtractor prior to span
    • Span B2
  • Span C
    • Span C1 // Set Span Attribute using Baggage (fails, baggage empty)
    • Span C2

In my attempts, the baggage is set correctly during the scope of Span B1 (and is propagated and available to any child spans), and is successfully propagated if any child spans perform client/producer operations. But the baggage disappears once that span scope is closed (by design). So in the context of span C1, the previous baggage values from B1 are not available.

Is there any idiomatic way to do this? Walking back up the scope/context tree to set baggage at a higher level (context of Span A)? Storing baggage somewhere so that it can be looked up and set for spans using a ContextCustomizer? Storing global trace baggage context in thread local?

[laurit]

but it seems that this only works if the baggage was added in a scope that is current or an ancestor.

correct.

Walking back up the scope/context tree to set baggage at a higher level (context of Span A)?

this is not supported by the default context implementation. Context api acts like a stack, you can add elements on top of the stack and pop from the top, there is no api to iterate the stack or insert elements anywhere but the top.

Storing baggage somewhere so that it can be looked up and set for spans using a ContextCustomizer

ContextCustomizer is part of the instrumenter api used by the agent. It can be used to add stuff to context before the span is started with instrumenter api. You can only add this to your own usages of instrumenter api, you can not modify/add a ContextCustomizer to other usages of the instrumenter api.

The typical way to do what you describe is to instrument something that runs early in the request processing, somewhere between creating the root span and creating the first span you wish to modify. Add a mutable object to the context (the default implementation of baggage is immutable). You can modify the state of that mutable object whenever you need and as you inserted it before any of the spans you wish to modify it won't go away when the span scope is closed.

[kbouck]

instrument something that runs early in the request processing, somewhere between creating the root span and creating the first span you wish to modify

Ok. I assume then that I could just detect whether a span is a root span, and add it then.

Add a mutable object to the context

Could you provide a small snippet of how to add this, and how to access it later? Assume it's some sort of Map to represent mutable global baggage (per trace).

If I want the contents of this mutable global baggage map to be accessible as actual baggage (values to be included in context propagation, etc), would it be appropriate to use a ContextCustomizer to merge the mutable global baggage map with whatever actual baggage exists in that scope? My ContextCustomer is only used for my own usages of the instrumented api.

[laurit]

To write and read values to context you use

// keys use identity comparison, you must use the same key instance when writing and reading
private static final ContextKey<T>  KEY = ContextKey.named("my-key");

Context newContext = context.with(KEY, value);
T t = newContext.get(KEY);

Alternatively you can use a class that implements ImplicitContextKeyed. Some samples https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/internal/HttpRouteState.java https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/grizzly-2.3/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/grizzly/GrizzlyErrorHolder.java

If I want the contents of this mutable global baggage map to be accessible as actual baggage (values to be included in context propagation, etc), would it be appropriate to use a ContextCustomizer to merge the mutable global baggage map with whatever actual baggage exists in that scope?

Yes, I believe it would.