Allow setting combinations of build options

[msimberg]

It would be useful to be able to pass a set of build options to the configs to easier run builds with combinations of options. These would be for example:

• compiler
• compiler version
• dependency (e.g. boost) version
• compiler/linker flags
• sanitizer
• and so on...

These can be generic though so that projects can define their own set of options. I'm not sure what the best syntax would be for this (could be command line options or a separate build options config file), but essentially one should be able to launch pycicle with:

• project=hpx
• machine=daint
• build_options={compiler: gcc49, clang5; generic_option_1: value_1, value_2; generic_option_2: value_1, value_2}

pycicle would then generate the cartesian product of the build options and launch a build for each combination with ctest -S config/hpx/daint.cmake ... -Dcompiler=gcc49 -Dgeneric_option_1=value_1 -Dgeneric_option_2=value_1.

The build name should be derived from the build options to make them unique.

Conflicts/dependencies between build options are project specific anyway, so this should be handled on a per project basis in the config files.

One may not want to run builds for all combinations of build options so one could additionally:

1. launch multiple instances of pycicle to piece together the combinations one wants (will this work now?), or
2. have a way to include or exclude specific combinations (probably easiest to put in a separate file which is passed to pycicle)

Ideally the build options and config (daint.cmake) would be taken from the source repository but this might require a bigger refactoring.

[biddisco]

Ideally the build options and config (daint.cmake) would be taken from the source repository but this might require a bigger refactoring.

I have created #14 to cover this point as I think it will be very useful in the long term.

[msimberg]

For sanitizer builds I set the following (in addition to the normal flags):

• CXXFLAGS=-fsanitize=address -fsanitize=leak -fno-omit-frame-pointer (or thread, undefined, memory; leakandaddress` can be enabled at the same time)
• LDCXXFLAGS=-fsanitize=address -fsanitize=leak (same as CXXFLAGS but without -fno-omit-frame-pointer)
• HPX_WITH_SANITIZERS=ON
• HPX_WITH_STACKOVERFLOW_DETECTION=OFF

[sithhell]

While playing around with pycicle, I found the most convenient way is to generate one cmake file per config, with specific base options for the machine. This avoids cluttering of the config files, and you have a nice overview of them as well by just doing a directory listing. The "machine" then isn't the machine anymore, but a specific config. The only thing necessary for that to happen in pycicle is to let the config set the build type it wants.

[biddisco]

That's the approach we're using at the moment and it works fine.

With HPX, we have so many configuration options that we can't possibly test all of them, so I wanted to create a matrix of combinations that pycicle would then select random combinations from and thereby allow many different configs to be tested.

What I'm curious about its just what combinations of builds we currently have on rostam to give me a start set of options to get going with.

[sithhell]

I am not a big fan of selecting configs randomly as this doesn't help with getting a reproducible set builds and some failures might just not get caught since we selected some strange option to begin with.
I'd rather go with the "most commonly used" ones.
In the end, the vast majority of different options is a UI problem of the dashboard, which should be tackled in the long run.

[biddisco]

The problem I want to address is that we get fails when APEX is enabled, but only in Debug mode and if boost>x and hwloc>y and that kind of thing. My plan was to allow pycicle to produce builds based on many options combined in random ways, but the exact combination used will always be available in the log. If you want to reproduce the same set of options again, then you will be able to comment on the PR in github and paste the options in the comment - thus forcing pycicle to use the options you've set rather than new random ones.
I agree that in general, random options will be a pain to manage, but the purpose of this issue is to collect ideas on what needs to be tested and we can certainly allow the 'most common' options or one set of builds, and random ones for another. I hope.