Context propagation not working with v1.7.0

[vimal3271]

I have a kafka producer and transformer app. I am using OTEL collector with Jaeger as backend.

JAVA_AGENT_OPTS=" $JAVA_AGENT_OPTS -Dotel.traces.exporter=otlp -Dotel.exporter.otlp.traces.protocol=grpc -Dotel.exporter.otlp.endpoint=http://otel-svc:4317"

I am able to see the traces of producer and consumer. But they are not connected with same traceid. This was working fine till v1.5.3.

Any configuration that I need to enable.

[trask]

hi @vimal3271! the receive span now is the parent of the consumer span, and the receive span has a link to the producer span.

there's an option that may support the prior behavior, can you try with -Dotel.instrumentation.common.experimental.suppress-messaging-receive-spans=true, and see if that works for you?

[trask]

Without -Dotel.instrumentation.common.experimental.suppress-messaging-receive-spans=true, the agent should still be propagating spans, but instead of consumer.parent -> producer, it should be using links:

• consumer.parent -> receiver
• receiver.link -> producer

Unfortunately many backends do not support/render links yet though, which is why we also provide the option -Dotel.instrumentation.common.experimental.suppress-messaging-receive-spans=true

[vimal3271]

Thank you @trask

[Asafb26]

for this case, what is the effect on the propagated baggage? will the consumer not going to receive the baggage sent by the producer?

[mateuszrzeszutek]

for this case, what is the effect on the propagated baggage? will the consumer not going to receive the baggage sent by the producer?

That's an interesting case, we haven't considered that. No, by default the consumer will not get the baggage sent by the producer - just the span link is preserved. It should all work if you set the -Dotel.instrumentation.common.experimental.suppress-messaging-receive-spans=true property though.

[wijnanjo]

So this new implementation (based on the current spec) makes it impossible (or hard) to correlate logs based on a common trace-id, right? In the case of a simple kafka producer and consumer, we get loglines with different trace-ids.

Do we then need backends capable of stitching together the spans based on the parents and links ? I might have it wrong but the new behavior seems to break logical flows.

[mvkrishna86]

Hi @trask , I am using Tempo/Grafana for storing and visualizing traces. I am having the same issue that the producer and consumer are having different traceids.
I tried adding -Dotel.instrumentation.common.experimental.suppress-messaging-receive-spans=true. But still I am getting the traceids differently.
I am using java agent 1.6.0 jar file (auto instrumentation). Can you suggest any alternative please?

[mvkrishna86]

Thanks @trask , I am planning to migrate to 1.9.0. I will create the issue if I cant see the issue fixed.

[mvkrishna86]

Its working with 1.9.0. Thanks @trask .

[sukrit007wawa]

Tried 1.9.1 with -Dotel.instrumentation.common.experimental.suppress-messaging-receive-spans=true and do not see traceid getting propogated with ADOT Java Auto instrumentation

From the trace, I can confirm: "otel.resource.process.command_line": "/usr/lib/jvm/java-11-openjdk:bin:java -javaagent:/work/aws-opentelemetry-agent.jar -Djava.awt.headless=true -Djava.awt.headless=true -Dotel.instrumentation.common.experimental.suppress-messaging-receive-spans=true"

[rashid4lyf]

Same here, tried with 1.9.0 and the traceid is not getting propagated. I can confirm from the logs that the flag is applied at runtime: java -javaagent:/app/newrelic/newrelic.jar -javaagent:/app/opentelemetry-javaagent.jar -Dotel.instrumentation.common.experimental.suppress-messaging-receive-spans=true -XX:+ExitOnOutOfMemoryError -Xms543m -Xmx543m -XX:MetaspaceSize=145m -XX:MaxMetaspaceSize=145m -jar /app/wh-kafka-standard.jar

[trask]

@sukrit007wawa @rashid4lyf can you open a github issue and provide a repro of the problem(s) you are seeing?

[PhoenixFlash10]

Do you have any documentation around this - -Dotel.instrumentation.common.experimental.suppress-messaging-receive-spans=true?
I couldn't find anything on https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/

@trask @vimal3271

[PhoenixFlash10]

@laurit thanks for your reply again
Does that mean if I set "otel.instrumentation.messaging.experimental.receive-telemetry.enabled" to true, it will copy the traceID and spanID from the producer?
or, I should set it to false and then will the traceID and spanID be copied?

[laurit]

Does that mean if I set "otel.instrumentation.messaging.experimental.receive-telemetry.enabled" to true, it will copy the traceID and spanID from the producer?

No it does not. It means that producer and consumer will be in separate traces that are connected with a span link. The default (receive-telemetry.enabled=false) has producer and consumer in the same trace.

[PhoenixFlash10]

can I get a few more details around how the instrumentation for Kafka messages work?
The reason for this question is - if there are multiple messages getting consolidated and read, how is the log tracing through Jaeger going to work?

[laurit]

can I get a few more details around how the instrumentation for Kafka messages work?

You can examine the source code of the instrumentation at https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/kafka/kafka-clients

The reason for this question is - if there are multiple messages getting consolidated and read, how is the log tracing through Jaeger going to work?

I guess it depends on what you mean by consolidating. Kafka instrumentation works best when you read a ConsumerRecord, act on it (i.e. do something that produces spans) and proceed to the next message. For example if you read a ConsumerRecord, put it into a list, do the same with other messages and finally process the list in a way that produces spans then it most likely won't work unless you manually restore the context for each ConsumerRecord.

[PhoenixFlash10]

ohh ok ok. This makes sense to me now. Let me check few more details around this and get back to you!