build system/CI: Support for Build Variants

[crasbe]

Description

Many examples and tests have configuration options that enable or disable certain parts and functionalities of the code.
If these code parts are guarded with preprocessor conditionals, some parts of the code will not be evaluated by the compiler.

This can lead to silent regressions that are not caught by our CI or even semi-automated testing from the dist/tools/compile_test/compile_like_murdock.py script.

Proposed Mechanism

The variants should be defined either in Makefile.ci or a separate Makefile.variants/Makefile.variants.ci file where configuration tuples (including but not limited to BOARD, USEMODULE, CFLAGS += -DCONFIG_...) are set.

These configuration tuples can be easily retrieved by running info-debug-variable-% and evaluated in the related scripts, e.g. .murdock or dist/tools/compile_test/compile_like_murdock.py (which calls .murdock).

I am not quite sure how the configuration will be applied, but it could look something like this for the mcp23x17 driver that supports both SPI and I2C:

CI_BUILD_VARIANT_LIST = \
  nrf52dk_with_i2c \
  nrf52dk_with_spi \
  #

ifeq (nrf52dk_with_i2c,$(CI_BUILD_VARIANT))
  BOARD = nrf52dk
  USEMODULE += mcp23x17_i2c
  USEMODULE += mcp23x17_reset
  CFLAGS += "-DMCP23X17_PARAM_RESET_PIN=GPIO_PIN(0,7)"

else ifeq (nrf52dk_with_spi,$(CI_BUILD_VARIANT))
  BOARD = nrf52dk
  USEMODULE += mcp23x17_spi
  USEMODULE += mcp23x17_reset
  CFLAGS += "-DMCP23X17_PARAM_RESET_PIN=GPIO_PIN(0,7)"

else
  ifeq (1,$(RIOT_CI_BUILD))
    $(error Unknown CI build variant: $(CI_BUILD_VARIANT))
  endif
endif

Since the parameters have to be set before calling the test/example Makefile, the Makefile.variants.ci could be included through the command line such as this for normal calls or for Docker:

CI_BUILD_VARIANT=nrf52dk_with_i2c MAKEFILES="Makefile.variants.ci" make

BUILD_IN_DOCKER=1 DOCKER_MAKE_ARGS="CI_BUILD_VARIANT=nrf52dk_with_i2c MAKEFILES=\"\$(DOCKER_APPDIR)Makefile.variants.ci\"" make

The latter could be hidden in the makefiles/docker.inc.mk, the former I'm not sure. For testing it would be nice to have the ability to actually run the variants, but I don't really see how else it would be possible to include the Makefile.variants.ci before the application Makefile is evaluated.

It would be possible to add a conditional in our build system that complains when CI_BUILD_VARIANT is set but MAKEFILES is not.

Integration into the CI

Currently I don't really understand at which point or how the CI gathers the jobs it can execute. As in: which part of which script has to be modified to read the CI_BUILD_VARIANTS script and how to add the variants to the job list.

Possible additional requirements (for variants that don't explicitly set a BOARD)

It is quite likely that some variants might require more memory than others, so either the BOARDS_INSUFFICIENT_MEMORY variable would have to be part of the Makefile.variants.ci (not very nice because that'll make it very hard to automate the way it currently is) or the make generate-Makefile.ci target has to be aware when the CI_BUILD_VARIANT is set and print a statement such as this:

Please add the following lines to the tests/sys/psa_crypto/Makefile.variants.ci for the CI_BUILD_VARIANT = fancy_ultralong_key:
BOARDS_INSUFFICIENT_MEMORY_ADDITIONAL = \
  nrf52dk \
  nucleo-l011k8 \
  #

[AnnsAnns]

probably makes sense to quickly mention this in the weekly