Opentelemetry objects in Heap

[AyushCarb1]

Hi Team,

We are observing that with OTEL instrumentation, the heap consumption by the application is higher as compared when the application is not instrumented.
With instrumentation , within 2 hours Heap reached upto ~1.5 GB , where as without instrumentation it reached ~ 750 mb .

On further checking the heap dump , we found lot objects from opentelemetry being occupied in memory which is having strong references and waiting for finalizer to be called.
GC algorithm used is Parallel GC.

Heap Dump:

Heap usage without instrumentation.

Heap usage with instrumentation

Can you please help us in understanding on object creation by otel instrumentation and retention of it in memory i.e. why there are so many objects created which is waiting for major GC to trigger to remove it from heap ?
Also, can you please suggest the best GC algorithm to be enabled for otel instrumented application.

[jkwatson]

moving to the instrumentation repository

[laurit]

On further checking the heap dump , we found lot objects from opentelemetry being occupied in memory which is having strong references and waiting for finalizer to be called.

Could you explain how you reached that conclusion. As far as I can tell the top half of the first screenshot says that there is a bit over 100mb of live objects and 30mb of garbage and and 260k of objects waiting for finalization. It does not show that these objects have anything to do with opentelemetry. The bottom half shows that there is a 4mb byte-buddy ExecutingTransformer and a bunch of other objects. My guess is that these other objects are reachable from the ExecutingTransformer and included in its 4mb size. 4mb isn't really much, or is it for you?
As for the increase in max heap size. If you don't limit the max heap size with -Xmx or if your -Xmx is larger than needed jvm will find itself a heap size that it feels comfortable with. It is reasonable to assume that it settles at a larger size because of the increased allocation rate or heap usage when running with the agent. The problem with these screenshots is that they do not really show that agent is using excessive amounts of memory, which it probably does not, but you can find this from the heap dump. To me the more likely cause would be that agent increases allocation rate. It would be helpful and would definitely improve the chances of this getting resolved if you could analyze this and provide us the information. I'd try flight recorder, if you are using jdk 17 the default configuration should do (it should include jdk.ObjectAllocationSample events) on older jdks you should use the profiling configuration (default one doesn't enable tlab allocation events). Based on these allocation events you should get a picture of the extra allocations that the agent causes and where they come from.