Calibrating CMOR3 & 4 forward development plans

[durack1]

updated to reflect backward compatibility for CMOR3.8

[Updated] Below is a roadmap to make project support (and timelines) apparent across planning CMOR releases.

The following is my current thinking on how this might look:

CMOR version	availability	project support	MIP table support	new features	repo
3.7.x	now	CMIP6_CVs	cmip6-cmor-tables	NA	PCMDI/cmor
3.8.x	March 2024	CMIP6_CVs, CMIP6Plus_CVs, input4MIPs_CVs, obs4MIPs_CVs	cmip6-cmor-tables, mip-cmor-tables	NA	PCMDI/cmor
3.9.x	~May 2024	CMIP6_CVs, CMIP6Plus_CVs, input4MIPs_CVs, obs4MIPs_CVs	cmip6-cmor-tables, mip-cmor-tables	new netcdf (quantize, standard), Apple Mx native support	PCMDI/cmor
4.0	~2025	CMIP6Plus_CVs, input4MIPs_CVs, obs4MIPs_CVs, CMIP7	mip-cmor-tables	pure python	new repo; separate dev allows ongoing 3.8/3.9 maintenance until CMOR4 feature complete

ping @matthew-mizielinski @piotr-florek-mohc @wolfiex @taylor13 @mauzey1

• consider if xCDAT functionality could be useful to reduce CMOR4 requirements?
• from a user-perspective, what is a reasonable timeline to assume the above versions would be supported until? In particular the CMOR3.7.x branch would be a good one to clarify

Anything else to consider, or tweak above?

[matthew-mizielinski]

It isn't immediately obvious to me that the changes for the mip-cmor-tables use will break CMOR for use of other sets of MIP tables, but I can see that maintaining the tests using the tables may be a little more tricky, and the tests in this repository would benefit from some reorganisation/attention.

Regarding the different forthcoming releases, I'd recommend creating and maintaining a "release branch" for version 3.7 while development proceeds on main for v3.8. A fix to 3.7 can then be applied to the release branch and released, and then it can be cherry-picked into main while v3.8 is being prepared. Once v3.8 is ready create a v3.8_release branch and any bug fixes to that version can proceed while developing v3.9.

The important thing with release branches is to ensure that a fix applied to the v3.7_release branch is propagated to main and other release branches if necessary. It is moderately straightforward to cherry-pick changes (providing changes to release branches are "squash and merge"'d) from one branch to another as long as the code has not diverged too far from the branch on which it is developed. It is also key that care is taken to try to keep a linear history within the repository as merges can interleave commits from one branch into another on merge

[durack1]

It isn't immediately obvious to me that the changes for the mip-cmor-tables use will break CMOR for use of other sets of MIP tables, but I can see that maintaining the tests using the tables may be a little more tricky, and the tests in this repository would benefit from some reorganisation/attention.

It's the formatting change strings to lists for dimensions and modeling_realm (e.g. CMIP6 Amon tas and MIP APmon tas) that breaks backward compatibility.

Of course, another option would be to reformat the cmip6-cmor-tables to align with CMOR3.8, but that would need to be thought about.

[matthew-mizielinski]

It isn't immediately obvious to me that the changes for the mip-cmor-tables use will break CMOR for use of other sets of MIP tables, but I can see that maintaining the tests using the tables may be a little more tricky, and the tests in this repository would benefit from some reorganisation/attention.

It's the formatting change strings to lists for dimensions and modeling_realm (e.g. CMIP6 Amon tas and MIP APmon tas) that breaks backward compatibility.

I would have thought it possible to run the equivalent of the following python code for the dimensions and modeling_realm

if isinstance(variable_entry['dimensions'], list):
    dimensions = variable_entry['dimensions']
else:
    dimensions = variable_entry['dimensions'].split(' ')

Of course, another option would be to reformat the cmip6-cmor-tables to align with CMOR3.8, but that would need to be thought about.

I wouldn't even consider this while CMIP6 is still operational

[mauzey1]

If the code that reads variable values from the tales encounters an array, then it will convert that array into a string.

cmor/Src/cmor_tables.c

Lines 215 to 233 in 10bb94a

	/* -------------------------------------------------------------------- */
	/* Attribute values that are arrays will have their array elements */
	/* combined into space-separated lists. */
	/* -------------------------------------------------------------------- */
	if(json_object_is_type(value, json_type_array)) {
	jsonArray = json_object_get_array(value);
	arrayLen = array_list_length(jsonArray);
	for (k = 0; k < arrayLen; k++) {
	jsonItem = (json_object *) array_list_get_idx(jsonArray, k);
	if (k == 0) {
	strcpy(szValue, json_object_get_string(jsonItem));
	} else {
	strcat(szValue, " ");
	strcat(szValue, json_object_get_string(jsonItem));
	}
	}
	} else {
	strcpy(szValue, json_object_get_string(value));
	}

Values for dimensions and modeling_realm will get treated as a space-separated list in a string regardless of their format in the tables.

[durack1]

@mauzey1 excellent, if I am reading (and interpreting this correctly) in then means that we can continue to use CMOR3.8 with the cmip6-cmor-tables, right?

[mauzey1]

Yes, CMOR 3.8 should still work with cmip6-cmor-tables.

[durack1]

@mauzey1 just circling around on this, and correcting the erroneous assumption that CMOR3.8/.9 was going to break backward (cmip6-cmor-tables) support, great.

Just trying to calibrate if anything is holding up the CMOR3.8 release, and if so, it'd be great to get any remaining issues identified so these can be resolved and CMOR3.8 can be released