Investigate general capabilities of REF

[alistairsellar]

Part of #280, follows #295

We should review the broader design and framework of the REF, and consider the option of consolidating on that framework if it can be made to meet the fundamental needs of model development, and whether sharing the framework is a more efficient approach overall. We should be prepared to reconsider our current plans.

Time box: 1-day

Anticipated output: Summary of key features that may be useful for CMEW, excluding those covered by #295, #333, #320 (ingest, solve, display). High level summary of what steps are involved in running it, and constraints on which ESMValTool recipes can be used and any work required to make a recipe REF-compatible. Link to any documentation on this. Recommend doing this in the form of a notebook demonstrating capabilities (there may be existing notebooks from the development).

Resources:

• REF introduction and context: https://wcrp-cmip.org/cmip-phases/cmip7/rapid-evaluation-framework/
• REF code and documentation: https://github.com/Climate-REF/climate-ref

[mo-nikosbaltas]

REF Documentation: https://climate-ref.readthedocs.io/en/latest/

This issue is actually a continuation of the investigation already done for #333, documenting some additional findings after tests carried out during #333.
There can be some repetition with #333 to make this complete.

Installation, configuration and running REF

The REF documentation (see above) seems to be the latest version, since REF is still under development.

Instructions on how to install, configure and run REF

cd ~
git clone https://github.com/Climate-REF/climate-ref.git
conda create -n climate-ref python=3.11
conda activate climate-ref
python -V  (should by 3.11)
pip install "climate-ref[aft-providers]"
ref --help

Create a REF workspace

mkdir ~/ref-workspace
cd ~/ref-workspace/
mkdir config

Set your username and export the valid path - the path is where you have created your ref-workspace

export REF_CONFIGURATION_DIRECTORY=/home/users/<username>/ref-workspace/config
ref config list > config/ref-config.toml

Ingest data to the catalogue
To ingest data to the catalogue, first make a CMEW run using both models (ref & eval)
After your run the data (cmip6) from the two model run should be in:
/home/users/<username>/cylc-run/CMEW/<runXX>/share/work/GCModelDev/ESMVal/MOHC
Note: For multi-runs (N>2) you need to run the parallel version of CMEW (#315, #338)

tree -L 3
.
├── HadGEM3-GC31-LL
│   └── amip
│       └── r5i1p1f3
├── HadGEM3-GC5E-LL
│   └── amip
│       └── r1i1p1f1
└── UKESM1-0-LL
    └── amip
        └── r1i1p1f1

In the case shown: [username=nikolaos.baltas, runXX=run51]
Check they exist.
/home/users/nikolaos.baltas/cylc-run/CMEW/run51/share/work/GCModelDev/ESMVal/MOHC
Now you ingest:
ref --log-level INFO datasets ingest --source-type cmip6 /home/users/<username>/cylc-run/CMEW/<runXX>/share/work/GCModelDev/ESMVal/MOHC

If everything went well, REF has indexed and stored the data in an SQLite database.
Check #333 for instructions (SQL queries) on how to view and extract the data from SQLite.
It also shows limitations of the REF CLI.

Overview and Findings from the investigation

Note that most of the statements related to REF were taken from their documentation.

Summary of key REF features relevant to CMEW
The Rapid Evaluation Framework (REF) is a governance and orchestration framework for climate model evaluation, designed to support large-scale, comparable, and reproducible diagnostic campaigns (e.g. CMIP-style assessments). It is not a replacement for CMEW or ESMValTool, but a higher-level framework that coordinates dataset cataloguing, diagnostic eligibility, execution, provenance tracking, and result curation.

---

Key features potentially useful for CMEW include:
• Persistent local dataset catalogue
REF ingests CMIP6-like datasets and stores rich scientific metadata and file paths in a local relational database. This enables efficient dataset discovery, grouping, and reuse without repeated filesystem scanning.
• Diagnostic orchestration (“solve” model)
REF determines which diagnostics are applicable to which datasets based on declared requirements, automatically grouping compatible datasets and managing execution at scale.
• Provider abstraction
REF delegates computation to diagnostic providers (e.g. ESMValTool, PMP, ILAMB), allowing multiple tools to be orchestrated under a common evaluation framework.
• Reproducibility and provenance
REF records which datasets, diagnostic definitions, and provider versions contributed to each result, supporting auditability and formal assessment workflows.
• Structured outputs (CMEC bundles)
Diagnostic outputs are packaged in standardised, machine-readable formats (metrics, plots, provenance), suitable for downstream visualisation or reporting.
These features align most strongly with comparative evaluation and assessment workflows, rather than routine development-time diagnostics.

---

High-level workflow: running REF with CMEW outputs
At a conceptual level, the REF workflow consists of the following steps:
1. Data production (outside REF)
CMEW runs development model configurations and standardises outputs into CMIP6-like NetCDF structure and metadata.
2. Dataset ingestion
REF ingests these outputs using ref datasets ingest --source-type cmip6, extracting and cataloguing metadata and file paths into a local SQLite database.
3. Dataset catalogue inspection
Users can query available datasets via the REF CLI (I found some limitations see #333) or directly via SQL for advanced use cases.
4. Diagnostic discovery and planning (ref solve --dry-run)
REF evaluates which diagnostics are applicable to the ingested datasets based on provider-declared requirements (variables, experiments, time coverage, etc.).
5. Diagnostic execution (ref solve)
REF orchestrates execution by invoking provider tools (e.g. ESMValTool) with the appropriate dataset groupings.
6. Result collection and inspection
Outputs (plots, metrics, data files, metadata) are stored in structured result directories and indexed by REF for inspection or visualisation.
In this model, CMEW remains responsible for producing standardised data, while REF manages what diagnostics run, on which datasets, and with what provenance.

---

Constraints and limitations: ESMValTool usage within REF
Constraints on ESMValTool recipes
REF does not support running arbitrary ESMValTool recipes out of the box. Instead:
• Only diagnostics explicitly registered with REF providers can be executed.
• Each diagnostic must declare:
o required variables,
o experiments,
o grouping logic,
o optional constraints (e.g. time coverage).
• Recipe configuration and flexibility are intentionally constrained to ensure comparability and reproducibility.
This differs from CMEW workflows, where users can freely select and modify ESMValTool recipes.

---

Current technical limitation (identified during investigation)
Although REF successfully ingests CMEW-standardised CMIP6-like data and stores full CMIP6 metadata in its database, the solver-stage catalogue projection currently exposes a limited, flattened set of metadata fields (see findings #333). As a result:
• The built-in ESMValTool provider expects CMIP6 facet names (variable_id, experiment_id, table_id) that are not exposed at solve time.
• This prevents ref solve from running ESMValTool diagnostics successfully in the current configuration.
This is a schema / integration mismatch, not a data quality issue, and would require either:
• enhancements to REF’s solver catalogue projection, or
• adaptation of the ESMValTool provider to the exposed schema.

---

Work required to make an ESMValTool recipe REF-compatible
To make ESMValTool diagnostics usable within REF, one of the following approaches is required:
1. Enhance REF core (longer-term)
Expose full CMIP6 facet metadata (variable_id, experiment_id, etc.) in the solver-stage catalogue so that existing provider requirements can be satisfied.
2. Adapt the ESMValTool provider (medium-term)
Modify provider requirements to use the flattened catalogue fields (e.g. experiment, instance_id) or parse CMIP6 identity from instance_id.
3. Introduce a CMEW-specific REF provider (pragmatic option)
Implement a CMEW-aware diagnostic provider that:
o understands CMEW-standardised outputs,
o selects datasets using currently exposed metadata,
o launches ESMValTool internally,
o produces REF/CMEC-compatible outputs.
The third option provides a realistic path to using REF orchestration without waiting for core REF changes.

---

Overall recommendation
• REF offers clear benefits for large-scale, comparative evaluation and formal assessment workflows, particularly where reproducibility, provenance, and comparability are critical.
• For routine CMEW development runs, REF does not currently provide additional scientific capability beyond CMEW + ESMValTool and introduces additional complexity.
• Consolidation on REF should be considered selectively, targeting campaign-style evaluation use cases rather than day-to-day development workflows.
• A CMEW-specific REF provider represents a practical integration path that preserves REF’s strengths while respecting CMEW’s existing capabilities.

---