Traces are not getting exported to Grafana for Java spring SOAP-WS services

[krishna-yalla]

Hi Team,

We are building SOAP-WS services using Java Spring Boot(3+).

I am trying to use -javaagent:opentelemetry-javaagent.jar for automatic instrumentation for traces in my project.

I have tried exported tracing in both ZipKin and Otlp format using opentelemetry-javaagent.

The docker config file is:

OTLP:

RUN wget -O opentelemetry-javaagent.jar 'https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar'
ENV JAVA_OPTS="-Xms4096m -Xmx4096m"
ENV GC_LOGS="-XX:+UseG1GC -XX:+UseAdaptiveSizePolicy -XX:+UseStringDeduplication -Xlog:gc=debug:file=/tmp/gc.log:time,uptime,level,tags:filecount=10,filesize=10m"
ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -javaagent:opentelemetry-javaagent.jar -Dotel.service.name=app-name
-Dotel.traces.exporter=otlp
-Dotel.exporter.otlp.endpoint=http://grafana-agent-host:4317
-Dotel.metrics.exporter=none
-Dotel.javaagent.debug=true -jar /usr/app/app.jar"]

Zipkin:

-javaagent:/path/opentelemetry-javaagent.jar
-Dotel.service.name=app-name
-Dotel.traces.exporter=zipkin
-Dotel.exporter.zipkin.endpoint=http://grafana-agent-host:9411/api/v2/spans
-Dotel.javaagent.debug=true
-Dotel.metrics.exporter=none

Agent logs(exporter=Otlp):
[otel.javaagent 2023-11-17 09:03:28:921 +0000] [http-nio-8080-exec-9] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'POST /service/ws/' : 6c1e8217051a484da91558aa7b48f636 fce9f3e9201dd59b SERVER [tracer: io.opentelemetry.tomcat-10.0:1.31.0-alpha] AttributesMap{data={http.method=POST, http.scheme=http, net.protocol.version=1.1, net.sock.peer.port=35015, thread.id=31, net.protocol.name=http, net.sock.host.port=8080, net.sock.host.addr=..., http.status_code=200, http.route=/service/ws/, http.response_content_length=570, thread.name=http-nio-8080-exec-9, http.client_ip=..., http.request_content_length=1236, http.target=/service/ws, net.sock.peer.addr=127.0.0.6, user_agent.original=Apache-HttpClient/4.5.5 (Java/16.0.2), net.host.name=k8s.host}, capacity=128, totalAddedValues=18}

Agent logs(exporter=zipkin):

[otel.javaagent 2023-11-22 12:09:32:240 +0000] [http-nio-8080-exec-3] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'POST /service/ws/' : 74f44c7b66fe84fe7304407c704b4917 b3f9b3f5922fd90f SERVER [tracer: io.opentelemetry.tomcat-10.0:1.32.0-alpha] AttributesMap{data={http.client_ip=..., thread.name=http-nio-8080-exec-3, net.host.name=k8s.host, user_agent.original=Apache-HttpClient/4.5.5 (Java/16.0.2), http.request_content_length=1236, http.target=/service/ws, net.sock.peer.addr=127.0.0.6, net.protocol.name=http, http.status_code=200, http.scheme=https, net.protocol.version=1.1, http.method=POST, http.response_content_length=570, net.sock.host.port=8080, thread.id=25, http.route=/service/ws/, net.sock.host.addr=...,, net.sock.peer.port=60101}, capacity=128, totalAddedValues=18}

I am seeing the following error when trying to view traces in the Grafana UI:

exporter=Otlp:

[otel.javaagent 2023-11-17 09:03:32:875 +0000] [OkHttp http://grafana-agent-host:4317/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message: Broken pipe

exporter=Zipkin:

[otel.javaagent 2023-11-22 12:09:33:542 +0000] [BatchSpanProcessor_WorkerThread-1] DEBUG io.opentelemetry.sdk.trace.export.BatchSpanProcessor - Exporter failed
[otel.javaagent 2023-11-22 12:09:33:542 +0000] [OkHttp http://grafana-agent-hostc:9411/...] WARN io.opentelemetry.exporter.zipkin.ZipkinSpanExporter - Failed to export spans

Note: I have installed Jaeger & Zipkin in my local docker and tested my Java application. Traces are getting exported through the Opentelemetry java agent (Automatic instrumentation) and are visible in my local Jaeger UI(for Otlp) and Zipkin UI(for Zipkin).

We have tried for rest service which is working as expected in local and dev also, seeing issue only for soap web service.

Could someone share some lights here , why traces are not exported to the Grafana agent for SOAP-WS services? The team confirmed that the Grafana agent is rightly configured with proper receivers.

Does Grafana agent have some limitations to receiving traces from SOAP-WS services developed in Spring boot?

Thank you.
Krishna Y

[laurit]

Do I understand correctly that when you call the soap service you get the POST /service/ws/' span? What did you expect to get?

[krishna-yalla]

The issue got fixed with the Opentelemetry agent configuration.

[nandha25git]

Could you share the configuration that fixed the issue