How a traceId created in opentelemetry?

[lmssy]

I'm using javaagent 1.4.1 to my server which is used as an API Gateway who receive http requests from clients and transfer to dubbo requests to backend services.

I found that many http requests received by my API Gateway, the traceIds generated by my API Gateway are the same, for example as you can see below chart, it's all the same traceId 00c0a803cb16288385756511029d0001 from 2021-08-22 ~ 2021-08-23 with different http request received, it's wired, so I want to know:

1. how a traceId is generated? UUID or something else? is there any document or code that I can see?
2. Why those request received in different thread have same traceId?

instrumentationLibrary.name is io.opentelemetry.javaagent.jetty-8.0
The API Gateway use "Jetty Continuation" framework.

[mateuszrzeszutek]

Hey @lmssy ,

1. Trace & span ids are generated by the RandomIdGenerator class;
2. Maybe whatever is sending all those requests always sends the same traceId in the HTTP headers? If possible, please check the headers of incoming and outgoing API gateway messages - depending on the propagator used, e.g. for W3C tracecontext check the traceparent header. I suspect that there might be a process that always sends HTTP requests with the same traceId, one generated at the application start for example.

[lmssy]

Thx for your reply,

All traceIds are generated by my API Gateway server, this server is the 'root' of a trace.
I'm wonding if there is something wrong with traceId propagating on the condition of using "Jetty Continuation" which is a Asynchronous Servlets framework, Jetty Continuation will suspend a http request and multiplexing current thread to process another request, as you can see details in this link https://wiki.eclipse.org/Jetty/Feature/Continuations.

Could you have a look or find someone to give a investigation on this? Thanks in advance！

[lmssy]

I have seen a comment above about how Jetty should handles async requests . please make sure jetty instrumentation do the right thing on this scene.

opentelemetry-java-instrumentation/instrumentation/servlet/servlet-common/library/src/main/java/io/opentelemetry/instrumentation/servlet/ServletHttpServerTracer.java

Line 30 in 7f37514

public abstract class ServletHttpServerTracer<REQUEST, RESPONSE>

[mateuszrzeszutek]

I'm wonding if there is something wrong with traceId propagating on the condition of using "Jetty Continuation" which is a Asynchronous Servlets framework, Jetty Continuation will suspend a http request and multiplexing current thread to process another request, as you can see details in this link https://wiki.eclipse.org/Jetty/Feature/Continuations.

That does sound like it maybe the cause behind your situation. We don't instrument Jetty continuations, and most likely all those continuation context switches and multiplexing are completely unnoticed by the OpenTelemetry agent and it assumes that everything is part of a single operation.

Could you have a look or find someone to give a investigation on this? Thanks in advance！

Can you create an enhancement issue for Jetty Continuation instrumentation? I'm afraid that most likely the maintainers of this project won't have time and resources to work on this, but maybe we'll find a volunteer that'll instrument this lib.

[lmssy]

Ok, I will create an enhancement issue, thank you for your help.