Doesn't support 1GB+ heapdumps

[fx1738]

When trying to find a leak in a 1.9G (nodeJS) heapsnapshot, it gets stuck at "calculating dominators and retained size"

It doesn't run out of memory either (I'm giving memlab CLI 12GB of mem, it uses 3.6 max)

Same issue is present in Chrome Dev Tools:

I've tried letting it just sit/run for hours, but I believe its crashing internally and not returning anything

[JacksonGL]

@FexileTV Would it be possible for you to share the 1.9GB snapshot? Or, as an alternative, could you get a smaller snapshot that still reproduces the problem?

[fx1738]

@FexileTV Would it be possible for you to share the 1.9GB snapshot? Or, as an alternative, could you get a smaller snapshot that still reproduces the problem?

The snapshot was taken from an production environment so it contains sensitive information which I'm not able to share.

The app uses .5GB under normal load (no memory leak) and when the rare leak does occur it instantly jumps to 3x the memory usage. So I'm unable to capture a smaller size heapsnapshot

[JacksonGL]

Without a repro of the issue, it will be challenging to identify the root cause and to validate the fix. The execution appears to stall at the dominator tree construction stage (it's not a crash, otherwise the JS runtime should exit).

I may implement a CLI option to bypass this particular step later. This could allow MemLab to work without the retained size data for each object within the heap. Consequently, it won't rank the memory leaks based on their potential size.

[fx1738]

I believe the issue could be reproduced by using any large heap snapshot.

The option to bypass the dominator tree might work, however I'm unsure how far that would get me to seeing whats causing the problem within my app.

I've managed to get a 1.3GB Heapsnapshot and let it run for 1 hour with the same issue:

[JacksonGL]

The current JavaScript heap parser is not very good at processing very large heap snapshots, since memory leaks could often be captured by much smaller snapshots in most of use cases. If your JavaScript heap takes 500MB under normal load, a potential optimization is to encode the memory-consuming part into ArrayBuffer whenever possible. (ArrayBuffer is seen as external memory to the JavaScript heap, it will cause less pressure to garbage collector and significantly reduce the heap dump size).

I guess your JavaScript heap is densely populated with objects, perhaps more so than usual. Consider trying the following steps to identify memory issues:

1. Comment out the body of this method:
   

   memlab/packages/core/src/paths/TraceFinder.ts

   Line 590 in 0f0517e

   calculateAllNodesRetainedSizes(snapshot: IHeapSnapshot): void {

2. Build memlab
3. Run your local build of memlab with the following command to find objects with the highest number of outgoing edges:

node packages/memlab/bin/memlab analyze object-fanout --snapshot <PATH/TO/YOUR/SNAPSHOT>

4. View the retainer trace of those objects:

node packages/memlab/bin/memlab view-heap --snapshot <PATH/TO/YOUR/SNAPSHOT> --node-id <HEAP-OBJECT-ID>

Edit: I'm still having trouble replicating the freeze that occurs while handling big heap snapshots. Every time I try, Node.js just runs out of memory.