Skip to content

ctsm5.4.024: Hot fix with updated derecho_intel and a few fixes for it #3796

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ekluzek merged 17 commits into from
Mar 12, 2026
+166 −120
Merged

ctsm5.4.024: Hot fix with updated derecho_intel and a few fixes for it #3796

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
1462e2e
Fix #3791 so will work with intel/2025.3.2 in ccs_config1.0.77, and c…
ekluzek Mar 4, 2026
8d995d2
Update to ccs_config1.0.77 so can make sure it's working
ekluzek Mar 4, 2026
f82bdae
Start on change files
ekluzek Mar 4, 2026
a5508c3
Merge branch 'master' into ccs_config_1077_derecho_intel_needed_updates
ekluzek Mar 4, 2026
580708d
Fix for #3799 so that lnd_tuning_mode can be used for the paramfile
ekluzek Mar 4, 2026
f2b311c
Add in the needed subdirectory
ekluzek Mar 4, 2026
4558ae2
Update parameter files to ones with new parameters added needed for t…
ekluzek Mar 9, 2026
d7ac582
Revert "Fix for #3799 so that lnd_tuning_mode can be used for the par…
ekluzek Mar 9, 2026
6c94193
Merge remote-tracking branch 'samsrabin/replace-more-netcdfs-with-ifx…
ekluzek Mar 9, 2026
04b9f0c
Merge branch 'master' into ccs_config_1077_derecho_intel_needed_updates
ekluzek Mar 11, 2026
ce0da53
Add some logging around the namelist read for ch4finundated streams, …
ekluzek Mar 11, 2026
3bd3f8a
Add failing tests on mpi-serial derecho_intel with the compiler updat…
ekluzek Mar 11, 2026
bc927c2
Remove the tests that aren't being done or that now pass
ekluzek Mar 11, 2026
2a43946
Add the two new expected fails
ekluzek Mar 11, 2026
7b34200
Final change files
ekluzek Mar 11, 2026
68ce70c
Another final update as some important notes were missing
ekluzek Mar 12, 2026
682b099
Real final change file updates, after review
ekluzek Mar 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ fxDONOTUSEurl = https://github.com/ESCOMP/mizuRoute
[submodule "ccs_config"]
path = ccs_config
url = https://github.com/ESMCI/ccs_config_cesm.git
fxtag = ccs_config_cesm1.0.75
fxtag = ccs_config_cesm1.0.77
fxrequired = ToplevelRequired
# Standard Fork to compare to with "git fleximod test" to ensure personal forks aren't committed
fxDONOTUSEurl = https://github.com/ESMCI/ccs_config_cesm.git
Expand Down
8 changes: 4 additions & 4 deletions bld/namelist_files/namelist_defaults_ctsm.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -623,10 +623,10 @@ attributes from the config_cache.xml file (with keys converted to upper-case).
<!-- The default filenames are given relative to the root directory
for the CLM2 data in the CESM distribution -->
<!-- Plant function types (relative to {csmdata}) -->
<paramfile phys="clm6_0" >lnd/clm2/paramdata/ctsm60_params.c260204.no_nan_fill.nc</paramfile>
<paramfile phys="clm6_0" lnd_tuning_mode="clm6_0_cam7.0">ctsm60-cam70_params.c260204.nc</paramfile>
<paramfile phys="clm5_0" >lnd/clm2/paramdata/clm50_params.c250311.no_nan_fill.nc</paramfile>
<paramfile phys="clm4_5" >lnd/clm2/paramdata/clm45_params.c250311.no_nan_fill.nc</paramfile>
<paramfile phys="clm6_0" >lnd/clm2/paramdata/ctsm60_params.c260303.nc</paramfile>
<paramfile phys="clm6_0" lnd_tuning_mode="clm6_0_cam7.0">lnd/clm2/paramdata/ctsm60-cam70_params.c260305.nc</paramfile>
<paramfile phys="clm5_0" >lnd/clm2/paramdata/clm50_params.c260305.nc</paramfile>
<paramfile phys="clm4_5" >lnd/clm2/paramdata/clm45_params.c260305.nc</paramfile>

<!-- ================================================================== -->
<!-- FATES default parameter file -->
Expand Down
2 changes: 1 addition & 1 deletion ccs_config
Open in desktop
Submodule ccs_config updated 2 files
+8 −0 machines/cmake_macros/nvhpc.cmake
+2 −2 machines/derecho/config_machines.xml
156 changes: 47 additions & 109 deletions cime_config/testdefs/ExpectedTestFails.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,31 +36,6 @@
</phase>
</test>

<test name="SMS_Ld5.f09_g17.IHistClm50Sp.derecho_intel.clm-nofire">
<phase name="SHAREDLIB_BUILD">
<status>FAIL</status>
<issue>#2784</issue>
</phase>
</test>
<test name="SMS_Ld5.f19_g17.IHistClm50Sp.derecho_intel.clm-nofire">
<phase name="SHAREDLIB_BUILD">
<status>FAIL</status>
<issue>#2784</issue>
</phase>
</test>
<test name="SMS_Ld5.f09_g17.IHistClm60Sp.derecho_intel.clm-nofire">
<phase name="SHAREDLIB_BUILD">
<status>FAIL</status>
<issue>#2784</issue>
</phase>
</test>
<test name="SMS_Ld5.f19_g17.IHistClm60Sp.derecho_intel.clm-nofire">
<phase name="SHAREDLIB_BUILD">
<status>FAIL</status>
<issue>#2784</issue>
</phase>
</test>

<!-- aux_clm test suite failures -->
<test name="ERP_Ld9.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdAllVars">
<phase name="COMPARE_base_rest">
Expand Down Expand Up @@ -88,46 +63,23 @@
<issue>#3454</issue>
</phase>
</test>
<test name="SMS_Ld2_D_PS.f09_g17.I1850Clm50BgcCropCmip6.derecho_intel.clm-basic_interp">
<phase name="RUN">
<status>FAIL</status>
<issue>#2787</issue>
<comment>The issue shows how to fix it.</comment>
</phase>
</test>
<test name="ERS_Ly3.f10_f10_mg37.I1850Clm50BgcCropCmip6.derecho_intel.clm-basic">
<phase name="RUN">
<status>FAIL</status>
<issue>#2787</issue>
<comment>The issue shows how to fix it.</comment>
</phase>
</test>
<test name="SMS_Lm1.f10_f10_mg37.I1850Clm50BgcCropCmip6waccm.derecho_gnu.clm-basic">
<phase name="RUN">
<status>FAIL</status>
<issue>#2787</issue>
<comment>The issue shows how to fix it.</comment>
</phase>
</test>

<test name="ERP_D_Ld9.f10_f10_mg37.I1850Clm60BgcCrop.izumi_nag.clm-clm60cam7LndTuningModeLDust">
<phase name="COMPARE_base_rest">
<test name="ERP_P64x2_Ld396.f10_f10_mg37.IHistClm60Bgc.derecho_intel.clm-monthly--clm-matrixcnOn_ignore_warnings">
<phase name="BASELINE">
<status>FAIL</status>
<issue>#3654</issue>
<comment>This is a problem with the nn mapalgo method in streams with ESMF prior to 8.9.0</comment>
<issue>#2619</issue>
<comment>This failure relates to the following REP failure.</comment>
</phase>
</test>

<test name="ERP_D_Ld9.f10_f10_mt232.IHistClm60BgcCrop.derecho_intel.clm-clm60cam7LndTuningModeLDust--clm-nnStreamMapAlgo">
<test name="ERP_D_P64x2_Ld3.f10_f10_mg37.I2000Clm50BgcCru.derecho_intel.clm-noFUN_flexCN--clm-matrixcnOn_ignore_warnings">
<phase name="COMPARE_base_rest">
<status>FAIL</status>
<issue>#3654</issue>
<comment>This is a problem with the nn mapalgo method in streams with ESMF prior to 8.9.0</comment>
<issue>#2619</issue>
<comment>This failure relates to the following REP failure.</comment>
</phase>
</test>

<test name="ERP_P64x2_Ld396.f10_f10_mg37.IHistClm60Bgc.derecho_intel.clm-monthly--clm-matrixcnOn_ignore_warnings">
<phase name="BASELINE">
<test name="ERP_D_P64x2_Ld3.f10_f10_mg37.I2000Clm50BgcCru.derecho_intel.clm-flexCN_FUN--clm-matrixcnOn_ignore_warnings">
<phase name="COMPARE_base_rest">
<status>FAIL</status>
<issue>#2619</issue>
<comment>This failure relates to the following REP failure.</comment>
Expand All @@ -148,87 +100,89 @@
</phase>
</test>

<test name="PEM_D_Ld9.ne30pg3_t232.I1850Clm60BgcCropG.derecho_intel.clm-clm60cam7LndTuningMode">
<phase name="COMPARE_base_modpes">
<test name="FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel">
<phase name="RUN">
<status>FAIL</status>
<issue>#2542</issue>
<issue>#3182</issue>
</phase>
</test>

<test name="SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop">
<test name="ERP_D_P64x2_Ld10.f10_f10_mg37.I2000Clm60Bgc.derecho_intel.clm-Hillslope">
<phase name="RUN">
<status>FAIL</status>
<issue>#1733</issue>
<issue>#3788</issue>
</phase>
</test>

<test name="FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel">
<!-- mpi-serial failures on derecho_intel with the intel compiler update to intel/2025.3.2 -->
<test name="ERP_Ld9.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdAllVars">
<phase name="RUN">
<status>FAIL</status>
<issue>#3182</issue>
<issue>#3798</issue>
<comment>Divide by zero happens when ch4finundatedmapalgo==bilinear with intel/2025.3.2, but passes for nn,consf,consd</comment>
</phase>
</test>

<test name="ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdST3">
<test name="ERS_D_Ld7_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropRs.derecho_intel.clm-decStart1851_noinitial.GC.ctsm5424acl_int">
<phase name="RUN">
<status>FAIL</status>
<issue>#2867</issue>
<issue>#3798</issue>
<comment>Divide by zero happens when ch4finundatedmapalgo==bilinear with intel/2025.3.2, but passes for nn,consf,consd</comment>
</phase>
</test>

<test name="ERP_D_P64x2_Ld10.f10_f10_mg37.I2000Clm60Bgc.derecho_intel.clm-Hillslope">
<test name="ERS_D_Mmpi-serial_Ld5.5x5_amazon.I2000Clm60FatesRs.derecho_intel.clm-FatesCold.GC.ctsm5424acl_int">
<phase name="RUN">
<status>FAIL</status>
<issue>#3788</issue>
<issue>#3798</issue>
<comment>Divide by zero happens when ch4finundatedmapalgo==bilinear with intel/2025.3.2, but passes for nn,consf,consd</comment>
</phase>
</test>

<!-- fates test suite failures -->

<test name="ERP_Ld9.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdAllVars">
<phase name="COMPARE_base_rest">
<test name="SMS_D_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm45BgcCropQianRs.derecho_intel.clm-cropMonthOutput.GC.ctsm5424acl_int">
<phase name="RUN">
<status>FAIL</status>
<issue>#3660</issue>
<comment>Restart issues with default "inactive" fields added to history by hist_all_fields.</comment>
<issue>#3798</issue>
<comment>Divide by zero happens when ch4finundatedmapalgo==bilinear with intel/2025.3.2, but passes for nn,consf,consd</comment>
</phase>
</test>

<!-- This is commented out because cime currently does NOT allow longnames in the XML in the XSD for this file
<test name="ERS_D_Ld30.f45_f45_mg37.HIST_DATM%CRUv7_CLM50%FATES_SICE_SOCN_SROF_SGLC_SWAV_SESP.izumi_nag.clm-FatesColdLandUse">
<test name="SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.derecho_intel.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO.GC.ctsm5424acl_int">
<phase name="RUN">
<status>FAIL</status>
<issue>#3500</issue>
<issue>#3798</issue>
<comment>Divide by zero happens when ch4finundatedmapalgo==bilinear with intel/2025.3.2, but passes for nn,consf,consd</comment>
</phase>
</test>
-->

<test name="ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesColdLUH2">
<test name="SMS_Lm3_D_Mmpi-serial.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdHydro.GC.ctsm5424acl_int">
<phase name="RUN">
<status>FAIL</status>
<issue>#2653</issue>
<issue>#3798</issue>
<comment>Divide by zero happens when ch4finundatedmapalgo==bilinear with intel/2025.3.2, but passes for nn,consf,consd</comment>
</phase>
</test>

<test name="ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLUH2HarvestEvent">
<test name="ERP_P256x2_Ld30.f45_f45_mg37.I2000Clm60FatesRs.derecho_intel.clm-mimicsFatesCold">
<phase name="RUN">
<status>FAIL</status>
<issue>FATES#1216</issue>
<issue>#2261</issue>
</phase>
</test>

<test name="SMS_D_Ld3.f09_g17.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdSatPhen_prescribed">
<phase name="RUN">
<!-- fates test suite failures -->

<test name="ERP_Ld9.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdAllVars">
<phase name="COMPARE_base_rest">
<status>FAIL</status>
<issue>#2321</issue>
<issue>#3660</issue>
<comment>Restart issues with default "inactive" fields added to history by hist_all_fields.</comment>
</phase>
</test>

<test name="ERP_P256x2_Ld30.f45_f45_mg37.I2000Clm60FatesRs.derecho_intel.clm-mimicsFatesCold">

<!-- This is commented out because cime currently does NOT allow longnames in the XML in the XSD for this file
<test name="ERS_D_Ld30.f45_f45_mg37.HIST_DATM%CRUv7_CLM50%FATES_SICE_SOCN_SROF_SGLC_SWAV_SESP.izumi_nag.clm-FatesColdLandUse">
<phase name="RUN">
<status>FAIL</status>
<issue>#2261</issue>
<issue>#3500</issue>
</phase>
</test>
-->

<test name="ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesColdPRT2">
<phase name="RUN">
Expand All @@ -252,20 +206,6 @@
</phase>
</test>

<test name="ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLandUse">
<phase name="SHAREDLIB_BUILD">
<status>FAIL</status>
<issue>#2810</issue>
</phase>
</test>

<test name="ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesColdLandUse">
<phase name="SHAREDLIB_BUILD">
<status>FAIL</status>
<issue>#2810</issue>
</phase>
</test>

<test name="PVT_Lm3.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesLUPFT">
<phase name="RUN">
<status>FAIL</status>
Expand All @@ -280,15 +220,13 @@
</phase>
</test>


<test name="ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLUH2HarvestArea">
<phase name="RUN">
<status>FAIL</status>
<issue>#3789</issue>
</phase>
</test>


<test name="ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLUH2HarvestMass">
<phase name="RUN">
<status>FAIL</status>
Expand Down
106 changes: 106 additions & 0 deletions doc/ChangeLog
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,110 @@
===============================================================
Tag name: ctsm5.4.024
Originator(s): erik (Erik Kluzek,UCAR/TSS,303-497-1326)
Date: Wed Mar 11 11:46:22 PM MDT 2026
One-line Summary: Update derecho_intel again and some fixes for it

Purpose and description of changes
----------------------------------

Update to ccs_config1.0.77 with derecho_intel updates to intel/2024.3.2. And a two line fix to a continuation line in
the middle of a string, so that it will build.
Note, there are several mpi-serial Bgc cases that now fail on derecho_intel see details below.

Also update the parameter files to ones that have some new fields on it that will be moved from namelist to the parameter
file in a future b4b-dev PR.

Improve the logging about the ch4finundation stream. Some of this seemed to be important in order to be able to change
ch4finundationmapalo away from bilinear.

Remove a bunch of tests in the expected fail files that have been working now. Add a few more newly failing tests.

Significant changes to scientifically-supported configurations
--------------------------------------------------------------

Does this tag change answers significantly for any of the following physics configurations?
(Details of any changes will be given in the "Answer changes" section below.)

[Put an [X] in the box for any configuration with significant answer changes.]

[ ] clm6_0

[ ] clm5_0

[ ] ctsm5_0-nwp

[ ] clm4_5


Bugs fixed
----------

List of CTSM issues fixed (include CTSM Issue # and description) [one per line]:
Fixes #3791 - one line change needed for intel/2025.3.2
Fixes #3795 - Update to ccs_config_cesm1.0.77

Notes of particular relevance for users
---------------------------------------

Caveats for users (e.g., need to interpolate initial conditions):

Some mpi-serial cases for single point cases with a Bgc compset fail because of a divide by zero in the ch4finundation mapping.
It fails with ch4finundatedmapalgo==bilinear but works if set to "nn" (see https://github.com/ESCOMP/CTSM/issues/3798)

Changes to the parameter file (output of tools/param_utils/compare_paramfiles):

./tools/param_utils/compare_paramfiles $CSMDATA/lnd/clm2/paramdata/ctsm60_params.c260204.no_nan_fill.nc $CSMDATA/lnd/clm2/paramdata/ctsm60_params.c260303.nc
File 0: /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/paramdata/ctsm60_params.c260204.no_nan_fill.nc
File 1: /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/paramdata/ctsm60_params.c260303.nc

Variable(s) present in File 1 but not File 0:
baseflow_scalar
precip_repartition_glc_all_rain_t
precip_repartition_glc_all_snow_t
precip_repartition_nonglc_all_rain_t
precip_repartition_nonglc_all_snow_t

Notes of particular relevance for developers:
---------------------------------------------
There are now more problems with the CN-Matrix threaded tests (see https://github.com/ESCOMP/CTSM/issues/2619)

Testing summary: regular
----------------

[PASS means all tests PASS; OK means tests PASS other than expected fails.]

regular tests (aux_clm: https://github.com/ESCOMP/CTSM/wiki/System-Testing-Guide#pre-merge-system-testing):

derecho ----- OK
izumi ------- OK

If the tag used for baseline comparisons was NOT the previous tag, note that here:


Answer changes
--------------

Changes answers relative to baseline: no bit-for-bit -- except the compiler update means derecho_intel changes answers

Summarize any changes to answers, i.e.,
- what code configurations: All
- what platforms/compilers: derecho_intel
- nature of change: should be roundoff (but haven't verified)

73 of the 280 passing tests on derecho_intel show different answers

Other details
-------------

List any git submodules updated (cime, rtm, mosart, cism, fates, etc.):
ccs_config to ccs_config_cesm1.0.77

Pull Requests that document the changes (include PR ids):
(https://github.com/ESCOMP/ctsm/pull)
#3796 -- Hot fix with updated derecho_intel and changes needed for it

===============================================================
===============================================================
Tag name: ctsm5.4.023
Originator(s): samrabin
Date: Tue Mar 10 07:44:59 PM MDT 2026
Expand Down
1 change: 1 addition & 0 deletions doc/ChangeSum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
Tag Who Date Summary
============================================================================================================================
ctsm5.4.024 erik 03/11/2026 Update derecho_intel again and a few fixes for it
ctsm5.4.023 samrabin 03/10/2026 Replace more netCDFs with NaN issues
ctsm5.4.022 erik 03/04/2026 Change to using the ifx compiler and some more fixes related to it
ctsm5.4.021 erik 02/20/2026 Bring b4b-dev to master
Expand Down
Loading
Loading