CUPID LDF

3) Running LDF and ILAMB with CUPiD for an existing case

Background:

The Land Diagnostics Framework (LDF) was shamelessly built on top of work in CAM to construct the the Atmosphere Diagnostics Framework (ADF). The implementation and development of LDF is very much a work in progress and contributions to improve the workflow, usability, and capabilities are welcome.

The International Land Model Benchmarking package (ILAMB) was built as part of the DOE's RUBISCO project and provides a way to compare model cases with various observations.

The LDF:

• Creates single variable timeseries and monthly climatological means for a defined periods
• Calculates tables with global sums (or means) for requested variables from climatological means
• Plots mean annual timeseries for requested variables
• Plots annual and seasonal mean contour plots (from climatological means)
• Plots regional climatologies and time series for defined regions and quantities.
• A sample output from LDF is in this link.

ILAMB:

• Can work on raw history files or single variable timeseries.
• Uses dictionaries to translate model variable names and units for comparison with observations.
• Can compare multiple cases at different spatial resolutions.
• Requires regular lat-lon grids, so history files from unstructured grids have to be regridded (currently not supported in CUPID).

LDF is included as an external to CUPiD in CUPiD/externals/LDF, and ILAMB is an external to CUPiD in tools/CUPiD/externals/ILAMB.

NOTE: LDF compares two cases and currently has limited observations available in regional climatology plots. It works when comparing two structured (e.g. f09) or unstructured (ne30) cases. No regridding is required for comparisons of unstructured cases!

---

LDF and ILAMB can be run in CUPiD via the following steps

In this tutorial you will:

1. Set up the CUPID configuration file
2. Generate LDF and ILAMB configuration files
3. Launch an interactive job
4. Run LDF and ILAMB
5. Run CUPID diagnostics and webpage
6. View the LDF and ILAMB webpage

NOTE This tutorial assumes that you've already cloned CTSM with CUPID enabled (in ${CUPID_ROOT} and have the CUPID conda environments installed.

---

1) Set up the CUPID configuration file

1a) Migrate to your CTSM checkout with CUPID and the land_only example

cd ${CUPID_ROOT}/examples/land_only
conda activate cupid-infrastructure

1b) Create a softlink for the config file.

This example will create a subset of LDF output for quicker analyses DO THIS for the tutorial

ln -s config_subset.yml config.yml

Alternatively, for the full LDF output on a BGC case

ln -s config_full.yml config.yml

1c) Check and Modify the config.yml file

You can customize your post processing by modifying the cases or time period being analyzed, variables being plotted and more. This is not needed for the purposed of this tutorial, but it's helpful to see what's in your CUPID config file, so have a look

---

2) Generate LDF and ILAMB configuration files

• This step uses CUPiD/helper_scripts/generate_ldf_config_file.py and CUPiD/helper_scripts/generate_ilamb_config_files.py, which automatically pulls information from the CUPID config.yml

• The helper scripts require LDF and ILAMB templates that we've provided

• The output will be ldf_config.yml that we can use to run LDF and two files needed to run ILAMB (model_setup.txt and a .cfg file).

  This example will run LDF on structured cases (e.g. f09 or 4x5 grid) and use a reduced set of BGC variables for ILAMB DO THIS for the tutorial

${CUPID_ROOT}/helper_scripts/generate_ldf_config_file.py \
  --cupid-config-loc . \
  --ldf-template ${CUPID_ROOT}/externals/LDF/config_clm_structured_plots.yaml \
  --out-file ldf_config.yml

${CUPID_ROOT}/helper_scripts/generate_ilamb_config_files.py \
  --cupid-config-loc . \
  --run-type BGC_subset \
  --cupid-root ${CUPID_ROOT}

This example will run LDF on unstructured cases (ne30 grid), but because it's assuming unstructured history files CANNOT be run with ILAMB

${CUPID_ROOT}/helper_scripts/generate_ldf_config_file.py \
  --cupid-config-loc . \
  --ldf-template ${CUPID_ROOT}/externals/LDF/config_clm_unstructured_plots.yaml \
  --out-file ldf_config.yml

To run full ILAMB analyses for a BGC case use this --run-type, when you run ILAMB (below) it will take about 90 minutes on the 4x5 resolution example provided.

${CUPID_ROOT}/helper_scripts/generate_ilamb_config_files.py \
  --cupid-config-loc . \
  --run-type BGC \
  --cupid-root ${CUPID_ROOT}

OR, for the full ILAMB output on an SP case: (not tested at this time)

${CUPID_ROOT}/helper_scripts/generate_ilamb_config_files.py \
  --cupid-config-loc . \
  --run-type SP \
  --cupid-root ${CUPID_ROOT}

• HINT: If you want to add more models to the ILAMB comparison, just add them to the model_setup.txt file that was just created.

Checkpoint #3

You've now created the configuration files to run LDF and ILAMB. Go ahead and spend some time to look over the YAML files created and see what they tell you is going to happen when run.

---

3) Launch an interactive job on Casper

This creates an interactive shell on Casper running in the tutorial queue, using the LMWG project account. It sets up for using 16 processors to run over up to 2 hours of time.

export PBS_ACCOUNT=P93300041
export QUEUE=tutorial@casper-pbs
qinteractive -l select=1:ncpus=16:mpiprocs=16:mem=100G \
  -l walltime=02:00:00 \
  -q $QUEUE \
  -A $PBS_ACCOUNT 

---

4) Run ILAMB and LDF

• NOTE 1: If an LDF_output dir already exists in CUPiD/examples/land_only, remove or rename it
• NOTE 2: LDF will generate it's own single variable timeseries and climatology plots in your scratch directory.
  • If you've already run LDF for a case with the same time periods as you're analyzing here it will reuse those files.
  • If you're running over a different time period here, however, LDF will get confused by the old files so it's better to remove them.
• NOTE 3: It does not matter what order you run ILAMB and LDF (there is no dependency between the two). We choose to run ILAMB first because even the smaller configuration takes about 30 minutes.

Set up your environment in the interactive job

module load conda
module load nco
conda activate cupid-analysis
export ILAMB_ROOT=../../ilamb_aux

Run ILAMB and LDF

This uses the MPI (Message Passing Interface) Library to run the job in parallel using the 16 processors you opened up in the interactive shell above.

# Run ILAMB
mpiexec ilamb-run --config ilamb_nohoff_final_CLM_BGC_subset.cfg --build_dir ILAMB_output/ --df_errs \
${ILAMB_ROOT}/quantiles_Whittaker_cmip5v6.parquet --define_regions ${ILAMB_ROOT}/DATA/regions/LandRegions.nc \
${ILAMB_ROOT}/DATA/regions/Whittaker.nc --regions global --model_setup model_setup.txt --filter .clm2.h0a

# Run LDF
../../externals/LDF/run_adf_diag ldf_config.yml

for the full ILAMB output on a BGC case:

mpiexec ilamb-run --config ilamb_nohoff_final_CLM_BGC.cfg --build_dir ILAMB_output/ --df_errs ${ILAMB_ROOT}/quantiles_Whittaker_cmip5v6.parquet --define_regions ${ILAMB_ROOT}/DATA/regions/LandRegions.nc ${ILAMB_ROOT}/DATA/regions/Whittaker.nc --regions global --model_setup model_setup.txt --filter .clm2.h0a

---

These steps will create directories for ILAMB_output and LDF_output in ${CUPID_ROOT}/examples/land_only

Remember, these tasks take a while. When running ILAMB you'll see a note that says:

Setting up model results from model_setup.txt

It make take > 5 minutes, but eventually it will start looking something is happening.

LDF gets going faster, but it will throw a bunch of errors, especially for the regional plotting.

• Why do you think this happening?

Checkpoint #4

At this point, everyone should be running LDF. We'll learn a little more about CUPiD (and these packages) while we wait for them to finish. You may need to periodically check on your run to make sure ILAMB gets launched after LDF finishes.

---

---

Next examining the CUPiD Config file