Investigate CMIP REF templating capabilities

[alistairsellar]

Part of #280

Investigate how CMEW performs the following tasks:

• templating ESMValTool recipes / adding new datasets to a template

Time box: half a day

Anticipated output: Summary of how REF performs the above tasks, to inform whether these are worth pursuing further. Potentially, recommendations of whether or how CMEW could make use of those REF capabilities. (E.g. comments in this issue, which packages it uses, a few sentences on how it achieves these capabilities, any reflections on compatibility or alignment with CMEW, accessibility / entry points / API.)

Resources:

• REF introduction and context: https://wcrp-cmip.org/cmip-phases/cmip7/rapid-evaluation-framework/
• REF code and documentation: https://github.com/Climate-REF/climate-ref

[zmaalick]

Hi,

I have investigated the REF framework. With respect to templating, this framework work differently, it produced the recipe or config files in the background not giving the access to users. Does not offer much freedom to user. After downloading it 'ingest' the data:

"Ingestion extracts metadata from your locally downloaded datasets and stores it in a local catalogue for easy querying and filtering. This makes subsequent operations, such as running diagnostics, more efficient as the system can quickly access the necessary metadata without needing to reprocess the files."

After ingest it run the solver that does diagnostics part. The general command run on all available data and try to perform all possible diagnostics, however, it is possible to select specific data and diagnostics.

# Run only PMP diagnostics
ref solve --provider pmp

# Run only diagnostics containing "enso" in their slug
ref solve --diagnostic enso`

If we are looking a compatibility with CMEW, I think we can replace esmvaltool block with REF, CDDS will still be needed.

But there are some limitations which needs to address. CMEW model id and suite id which might not be possible in REF. Differentiating different model run might be problematic and might need to re-design the workflow.

As a next step, I would vote for creating a demo notebook to better access the capabilities and limitations wrt CMEW.

Doc Source: https://climate-ref.readthedocs.io/en/latest/

[alistairsellar]

Thanks Zubair Maalick (@zmaalick). I need to digest this, but looks useful.

Naomi Parsons (@NParsonsMO) the catalogue bit of this could be relevant to #130

[alistairsellar]

Follow-up questions:

• How does REF update or write recipes in the solve step - is it writing a new recipe or updating a template / old recipe?
• Where does it write the recipes that it runs for the second part of the solve step?

Time box: 0.5 day

[zmaalick]

Below are some key findings:

Climate-ref uses ESMValTool version: 2.13.0

About Recipes:
It load the default recipes, update it and saved it for process.

       recipe = load_recipe(self.base_recipe)
        self.update_recipe(recipe, input_files)
        recipe_txt = yaml.safe_dump(recipe, sort_keys=False)
        logger.info(f"Using ESMValTool recipe:\n{recipe_txt}")
        recipe_path = definition.to_output_path("recipe.yml")
        with recipe_path.open("w", encoding="utf-8") as file:
            file.write(recipe_txt)
        return recipe_path

So REF is using “structured templating”: load YAML → mutate dict → write YAML

Here is the file github file path:
https://github.com/Climate-REF/climate-ref/blob/main/packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/base.py

What’s happening step-by-step

• Loads a base ESMValTool recipe YAML into a Python dict.
• In REF terms: this base file is a “template” recipe.
• self.update_recipe This is the templating step.
• REF modifies the loaded YAML dict in memory:
• fills in the datasets: section based on input_files it has selected during “solve”
• may adjust diagnostics: settings, timersange, variables, preprocessors, etc.
• yaml.safe_dump(...) Dumps the now-updated dict back to YAML text.
• Write to recipe.yml
• That recipe.yml is what REF then runs with ESMValTool.

[ehogan]

Thanks Zubair Maalick (@zmaalick) 👍 I'd be really keen to understand what happens during the "mutate dict", specifically:

• self.update_recipe This is the templating step.
• REF modifies the loaded YAML dict in memory:
  • fills in the datasets: section based on input_files it has selected during “solve”
  • may adjust diagnostics: settings, timersange, variables, preprocessors, etc.

How is the REF filling in the datasets and adjusting diagnostics? What specific "templating" is occurring here? 🤔

[zmaalick]

I have been able to run solve.

Sample CMIP dataset I used are residing at:
/data/scratch/zubair.maalick/ref/configuration/datasets/sample-data
and results can be explored at:
/data/scratch/zubair.maalick/ref/configuration/results

the recipe each diagnostics run can be viewed in out.log and visualisation can be viewed in plots dir.

Looking into recipe, input data and output plots. I am not sure if the behaviour is as expected as I expect more visualisations but almost in every case have one.