Record memory usage

[adam-sim-dev]

Dear developers,

Can Caliper record the memory usage versus time?

Best,
Adam

[daboehme]

Hi @adam-sim-dev ,

One relatively straightforward way to get over-time measurements is Caliper's iteration instrumentation and loop profiling feature, as shown here: https://software.llnl.gov/Caliper/CaliperBasics.html#loop-profiling

You can then combine that with options to record memory usage. There are basically two: mem.highwatermark shows a high-watermark of all memory allocated through malloc(). Caveat is that it is relatively high-overhead (it intercepts every malloc call). It also won't capture memory allocated in other ways (e.g., cudaMalloc()) unless they fall back on malloc.

Another option is mem.pages, which records the virtual memory size and RSS as reported by Linux. It's not perfectly accurate but typically lower-overhead, and should at least give you a rough idea if there are memory leaks etc.

If you're using Umpire for memory management then Caliper can also record Umpire memory pool statistics.

Here an example of a loop profile with memory high-watermark info for Caliper's cxx-example program:

$ CALI_CONFIG=loop-report,mem.highwatermark,iteration_interval=10 ./examples/apps/cxx-example 100

Loop summary:
------------

Loop     Iterations Time (s) Iter/s (min) Iter/s (max) Iter/s (avg) Allocated MB 
mainloop        100 0.016529     0.000000  6152.566420  6049.961428     0.000072 

Iteration summary (mainloop):
-----------------

Block Iterations Time (s) Iter/s      Allocated MB 
               0 0.000004    0.000000     0.000072 
    0         10 0.001673 5977.282739     0.000072 
   10         10 0.001664 6009.373421     0.000072 
   20         10 0.001630 6136.068547     0.000072 
   30         10 0.001646 6076.312408     0.000072 
   40         10 0.001667 6000.178805     0.000072 
   50         10 0.001660 6025.403099     0.000072 
   60         10 0.001636 6111.173241     0.000072 
   70         10 0.001634 6120.505407     0.000072 
   80         10 0.001625 6152.566420     0.000072 
   90         10 0.001690 5916.088566     0.000072 

There's not much of a change in the high-watermark as this code doesn't allocate anything inside the loop, but this is how the output would look like. Is this what you're looking for?

[adam-sim-dev]

Another option is mem.pages, which records the virtual memory size and RSS as reported by Linux. It's not perfectly accurate but typically lower-overhead, and should at least give you a rough idea if there are memory leaks etc.

This option may help. I will have a try. Thanks.

[adam-sim-dev]

Hi @daboehme, what I want to do is like:

char* caliper_config = "mem.pages";

CALI_MARK_BEGIN("My code block");

# My code block

CALI_MARK_END("My code block");

to record the memory usage for the code block.

From https://github.com/LLNL/Caliper/blob/165b89ad00d5b235bfdae7215df7afd9a820db0c/src/services/memusage/MemStatService.cpp#L96

It seems that only one process is recorded. I am using multiple processes with MPI (on a shared memory machine if that matters), so I want the memory usage for all the processes used. How could I get that information using Caliper?

Best,
Adam