Out of memory in case of very frequent hooked function

[alexkalmuk]

When the hooked function is called, let's say, 1 million times per second, the Orbit app consumes almost 5 Gb of RAM after approximately 10 seconds. If the frequency is lower (100.000 times per second), then the time extends proportionally. I suspect that it might happen because of the data collected for a histogram, but I'm not sure yet.
I'm wondering if it's an expected behavior or a bug?

[beckerhe]

Not sure if I would call it a bug, but it also doesn't surprise me. AFAIK we never optimized for such a high load of calls.

With "Orbit app" I guess you mean the client, right? So far the limiting factor has always been the rendering.

We calculate some histograms but I don't know from the top of my head when this happens. I know some parts are done in the postprocessing step, but probably not all of it.

Maybe @florian-kuebler knows more.

There are 2 ways to find out more details:

1. Orbit has introspection which can be enabled when Orbit is started with the --devmode option. (There is a button above the capture window.). This allows you to see how long certain manually instrumented function in the Orbit Client take to execute. So far mostly the rendering code is instrumented.
2. You can use Orbit to profile Orbit. Therefore you have to start both OrbitService and Orbit a second time. I assume you're running everything on the same machine. To make the two OrbitService not clash you have to give it a different port number. Hence start the second OrbitService with the options --grpc_port=1025 --producer_side_server=false and the client with the option --grpc_port=1025. Then you can choose the first OrbitService as the target process in the second instance of the client. (I hope that's not too confusing). (Note that you can't use user space instrumentation and/or manual instrumentation in the second Service/Client/instance-pair.

I hope that helps. 😬

[alexkalmuk]

Thank you for these hints! I'm not sure if it helps with memory profiling, but I'll give it a try in any case.
Yes, by "Orbit app" I meant the client.

[florian-kuebler]

Sorry for being late to this. So I am not surprised if we do waste (or even leak) memory at some places. We unfortunately did not had time to look into memory. Also I agree that profiling Orbit with Orbit (or using introspection), will work, but also that Orbit is not particularly good in memory profiling (it has some memory tracking though).

What I know for sure is, that we keep every record of a function call (what we call a timer) in memory. That would 1.000.000* 10sec * ~4x8 Bytes (uint64) is around 0.3 Gb. That does not explain that completely. But we do have some of these places. So I would not be surprised if this is already the reason.

Unfortunately, I won't have any time soon to look into this more closely....