Add optional eager program compilation via ICON4PY_EAGER_PROGRAM_COMPILATION

[msimberg]

🤖 Opened by an agent on behalf of @msimberg.

ICON4Py currently compiles field operators and programs lazily on first use. This shifts compilation overhead into the first timestep and makes timing less predictable.

This PR adds the ICON4PY_EAGER_PROGRAM_COMPILATION environment variable. When set to true, static field factories eagerly compile their programs during model setup instead of waiting for the first execution. This can front-load compilation cost and make subsequent timesteps more uniform. The default value in CI is set to true so GPU and MPI integration runs exercise the eager path.

Open questions:

• Local CPU integration tests did not show a measurable speed-up. Whether the eager compilation path improves runtime on GPU or in MPI configurations is still an open question that the CSCS CI runs should answer.

[msimberg]

cscs-ci run default;SESSIONS=model;MODEL_SUBPACKAGES=common:standalone_driver;MODEL_SUBSETS=integration;LEVELS=integration;BACKENDS=gtfn_gpu;GT4PY_BUILD_CACHE_LIFETIME=session;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;SESSIONS=model;MODEL_SUBPACKAGES=common:standalone_driver;MODEL_SUBSETS=integration;LEVELS=integration;BACKENDS=dace_gpu;GT4PY_BUILD_CACHE_LIFETIME=session;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=common:standalone_driver;MODEL_MPI_SUBSETS=integration;LEVELS=integration;BACKENDS=gtfn_gpu;GT4PY_BUILD_CACHE_LIFETIME=session;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=common:standalone_driver;MODEL_MPI_SUBSETS=integration;LEVELS=integration;BACKENDS=dace_gpu;GT4PY_BUILD_CACHE_LIFETIME=session;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;SESSIONS=model;MODEL_SUBPACKAGES=common:standalone_driver;MODEL_SUBSETS=datatest;LEVELS=integration;BACKENDS=gtfn_gpu;GT4PY_BUILD_CACHE_LIFETIME=session;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;SESSIONS=model;MODEL_SUBPACKAGES=common:standalone_driver;MODEL_SUBSETS=datatest;LEVELS=integration;BACKENDS=dace_gpu;GT4PY_BUILD_CACHE_LIFETIME=session;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=common:standalone_driver;MODEL_MPI_SUBSETS=datatest;LEVELS=integration;BACKENDS=gtfn_gpu;GT4PY_BUILD_CACHE_LIFETIME=session;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=common:standalone_driver;MODEL_MPI_SUBSETS=datatest;LEVELS=integration;BACKENDS=dace_gpu;GT4PY_BUILD_CACHE_LIFETIME=session;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest;ICON4PY_EAGER_PROGRAM_COMPILATION=false

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest;GT4PY_BUILD_CACHE_LIFETIME=session;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest;GT4PY_BUILD_CACHE_LIFETIME=session;SLURM_PARTITION=normal;SLURM_TIMELIMIT=02:00:00;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest;GT4PY_BUILD_CACHE_LIFETIME=session;SLURM_PARTITION=normal;SLURM_TIMELIMIT=02:00:00;ICON4PY_EAGER_PROGRAM_COMPILATION=false

A/B comparison: the eager=false leg, matching the earlier eager=true run on the same config (from-scratch compile via GT4PY_BUILD_CACHE_LIFETIME=session).

🤖 Posted by an agent on behalf of @msimberg

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest;GT4PY_BUILD_CACHE_LIFETIME=session;SLURM_PARTITION=normal;SLURM_TIMELIMIT=02:00:00;GT4PY_BUILD_JOBS=32;ICON4PY_EAGER_PROGRAM_COMPILATION=true

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest;GT4PY_BUILD_CACHE_LIFETIME=session;SLURM_PARTITION=normal;SLURM_TIMELIMIT=02:00:00;GT4PY_BUILD_JOBS=32;ICON4PY_EAGER_PROGRAM_COMPILATION=false

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest

[msimberg]

cscs-ci run default;BACKENDS=dace_gpu;LEVELS=unit;MODEL_SUBPACKAGES=standalone_driver;SESSIONS=model;MODEL_SUBSETS=datatest

[msimberg]

cscs-ci run default;SESSIONS=model_mpi;BACKENDS=dace_gpu;LEVELS=integration;MODEL_MPI_SUBPACKAGES=standalone_driver;MODEL_SUBSETS=datatest

[msimberg]

cscs-ci run default;SESSIONS=model_mpi;BACKENDS=dace_gpu;LEVELS=integration;MODEL_MPI_SUBPACKAGES=standalone_driver;MODEL_SUBSETS=datatest

[msimberg]

cscs-ci run default;SESSIONS=model_mpi;BACKENDS=dace_gpu;LEVELS=integration;MODEL_MPI_SUBPACKAGES=standalone_driver;MODEL_SUBSETS=datatest

[msimberg]

cscs-ci run default;SESSIONS=model_mpi;BACKENDS=dace_gpu;LEVELS=integration;MODEL_MPI_SUBPACKAGES=standalone_driver;MODEL_SUBSETS=datatest

[github-actions]

When developing, you can test your changes on CSCS CI before merge with the default pipeline: cscs-ci run default. This will run a default subset of tests.

You can pass options to override pipeline variables, for example:

• cscs-ci run default;BACKENDS=gtfn_cpu;LEVELS=unit
• cscs-ci run default;MODEL_SUBPACKAGES=common:standalone_driver;SESSIONS=model
  Avoid running the pipeline for all tests when you are developing.

Available options are:

• SESSIONS: model, model_mpi, or tools (correspond to nox sessions)
• MODEL_SUBSETS: datatest, basic, or stencils (correspond to nox session selections)
• MODEL_SUBPACKAGES: subpackages for non-MPI tests (last component, e.g. diffusion, standalone_driver)
• MODEL_MPI_SUBPACKAGES: subpackages for MPI tests (as above)
• BACKENDS: backends
• GRIDS: grids for stencil tests (simple, icon_regional, or icon_global)
• LEVELS: testing level for non-stencil tests (unit or integration)

For each option, all can be used as a shorthand for all possible values of that variable, e.g. LEVELS=all.

See scripts/python/generate_ci_pipeline.py and noxfile.py for available values for each option.

The all pipeline can be run with cscs-ci run all. This will run all icon4py tests in CSCS CI which can be expensive. This pipeline runs on a schedule on main, and can be run when extensive validation is needed (e.g. before releases).

Merging

Once your PR is approved and ready for merging, add it to the merge queue. The merge CSCS CI pipeline will run automatically on the merge-queue branch and must pass before the PR is merged. A dummy merge check will be triggered on the PR itself since it's required to add a PR to the merge queue.

Optional Tests

To run benchmarks you can use:

• cscs-ci run benchmark-bencher

For more detailed information please look at CI in the EXCLAIM universe.

[msimberg]

cscs-ci run default;SESSIONS=model_mpi;BACKENDS=dace_gpu;LEVELS=integration;MODEL_MPI_SUBPACKAGES=standalone_driver;MODEL_SUBSETS=datatest