Using figure_of_merit_context with multiple lines of output

[amklinv-nnl]

I am trying to add a new package for SNAP and am grateful for the help y'all have provided so far. I am now able to successfully run my benchmark, but I'm struggling to capture the output in the way I'd like. I ran HPL and ramble workspace analyze produced the following output

From Workspace: hpl_test (hash: 717093d6d735577adab4292ad1456ab22648cccce537b1dccc084836e6b89d13)
Experiment hpl.standard.arbitrary_name figures of merit:
  Status = SUCCESS
  N-NB-P-Q = 500-192-2-2 figures of merit:
    Time = 0.00 s
    GFlops = 2.5272e+01 GFLOP/s
  N-NB-P-Q = 500-384-2-2 figures of merit:
    Time = 0.01 s
    GFlops = 1.3972e+01 GFLOP/s
  N-NB-P-Q = 1000-192-2-2 figures of merit:
    Time = 0.01 s
    GFlops = 6.4734e+01 GFLOP/s
  N-NB-P-Q = 1000-384-2-2 figures of merit:
    Time = 0.02 s
    GFlops = 3.7973e+01 GFLOP/s
  N-NB-P-Q = 500-192-1-4 figures of merit:
    Time = 0.00 s
    GFlops = 2.8448e+01 GFLOP/s
  N-NB-P-Q = 500-384-1-4 figures of merit:
    Time = 0.00 s
    GFlops = 1.9191e+01 GFLOP/s
  N-NB-P-Q = 1000-192-1-4 figures of merit:
    Time = 0.01 s
    GFlops = 6.9491e+01 GFLOP/s
  N-NB-P-Q = 1000-384-1-4 figures of merit:
    Time = 0.02 s
    GFlops = 3.8291e+01 GFLOP/s
  N-NB-P-Q = 500-192-4-1 figures of merit:
    Time = 0.00 s
    GFlops = 1.9776e+01 GFLOP/s
  N-NB-P-Q = 500-384-4-1 figures of merit:
    Time = 0.01 s
    GFlops = 1.2417e+01 GFLOP/s
  N-NB-P-Q = 1000-192-4-1 figures of merit:
    Time = 0.01 s
    GFlops = 4.8506e+01 GFLOP/s
  N-NB-P-Q = 1000-384-4-1 figures of merit:
    Time = 0.02 s
    GFlops = 2.7993e+01 GFLOP/s

I would like to do something similar for SNAP, but the process grid unfortunately is output over multiple lines.

  Parallelization Parameters
    npey =     4
    npez =     4
    nthreads =    2

This may just be a regular expression issue, but I can't for the life of me figure out how to capture that info for figure_of_merit_context. Can you please provide guidance?

[douglasjacobsen]

@amklinv-nnl Do you need it to be a context? IIRC SNAP doesn't print out multiple parallelization parameters in a given execution, so it is set explicitly for each ramble experiment.

In HPL, you give it a range of N, NB, P, and Q values, and HPL itself sweeps over all of them (which is where the context stuff comes in).

However, currently, contexts can only be defined by a specific line. Can you share the full output that you're looking to parse and I might have more useful feedback for you?

[rfbgo]

Hey just wanted to follow up on this too, as I'd like to understand it better. A few details our end:

What your requesting certainly seems very reasonable, and it's something we've talked about before but never had a super concrete use case.

The idea of a context today is more like "capture things that change within a run and give a way to group stuff that emits multiple values per run". Similar to what Doug said, I think something like they way you parameterized your decomp/parallel run/etc is likely not a context (and the HPL example is a bit misleading about that)

For something likestreamc we capture "number of threads" as a FOM, since it describes the run (but not as a context, since it will not vary within a run)

When we do a scaling study we change things like npey, but that is across different experiments and is separated at that level

@amklinv-nnl with all the above in mind, does SNAP have a good example of needing multi line contexts, or do you think capturing things like npey as a fom is OK? If we want to go the contexts route it would be super useful to have an example of the raw output so we can work on a design to support it