OpenTelemetry overhead in microservices architecture

[kes2464]

I have microservices architecture, and it doesn't happen for all services, but found some weirdness between 2 java services.

Service A (Spring) calls Service B (Jersey) through http/1.1 using okhttp client and when only Service B is instrumented, there's no issue (slight increase in latency but might be expected). However as soon as I turn instrumentation on on Service A as well, the latency of Service B increases a lot.

So Service B makes AWS DynamoDB calls, which are usually done within around 4ms, but increase to 20-30 ms.
Then also the total response time increases from 40-50ms to 150-200ms.

Turning it on Service A while keeping it off for Service B also does not have an issue. The throughput of those services are around 100K requests per minute.

I was checking the raw http request from Service A to B, and couldn't find anything extra other than traceparent header.

Any idea on what I should check?

[kes2464]

A few more observations.

So I have 2 other services. Service C and Service D.

Service C also uses DynamoDB, but a different table. Has opentelemetry agent enabled
Service D also uses DynamoDb, but another table. Has opentelemetry agent disabled.

Service C and D have no dependencies to A and B.

Service C is running fine with opentelemetry agent enabled, until Service A enables OpenTelemetry Agent.
Then Service C's latency to DynamoDB also increases, while Service D's latency stays stable.

From AWS Cloudwatch, DynamoDB metrics are stable, and seem very normal throughout the period.

Same thing is happening to Redis (Elasticache)