Divide by zero seen in update to intel/2025.3.2 compiler update

[ekluzek]

Updating to intel/2025.3.2 with the update to ccs_config1.0.77 shows a new divide by zero problem for five mpi-serial tests.

The tests that fail:

ERS_D_Ld7_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropRs.derecho_intel.clm-decStart1851_noinitial     (RUN)
ERS_D_Mmpi-serial_Ld5.5x5_amazon.I2000Clm60FatesRs.derecho_intel.clm-FatesCold  (RUN)
SMS_D_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm45BgcCropQianRs.derecho_intel.clm-cropMonthOutput        (RUN)
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.derecho_intel.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO       (RUN)
SMS_Lm3_D_Mmpi-serial.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdHydro 

Do so because of a divide by zero that looks like this:

forrtl: error (73): floating divide by zero
Image              PC                Routine            Line        Source             
libc.so.6          000014FBD7847900  Unknown               Unknown  Unknown
libesmf.so         000014FBD9E4B616  _ZN5ESMCI4BBoxC1E     Unknown  Unknown
libesmf.so         000014FBDA2CF410  _ZN5ESMCI16OctSea     Unknown  Unknown
libesmf.so         000014FBDA2D1568  _ZN5ESMCI9OctSear     Unknown  Unknown
libesmf.so         000014FBDA0C9F01  _ZN5ESMCI6InterpC     Unknown  Unknown
libesmf.so         000014FBDA239978  _ZN5ESMCI6regridE     Unknown  Unknown
libesmf.so         000014FBDA2783B8  _Z19ESMCI_regrid_     Unknown  Unknown
libesmf.so         000014FBDA1F8EC3  _ZN5ESMCI7MeshCap     Unknown  Unknown
libesmf.so         000014FBDA2C16B2  c_esmc_regrid_cre     Unknown  Unknown
libesmf.so         000014FBDA974A50  esmf_regridmod_mp     Unknown  Unknown
libesmf.so         000014FBDA6E7578  esmf_fieldregridm     Unknown  Unknown
cesm.exe           00000000037EA55C  shr_strdata_init          639  dshr_strdata_mod.F90
cesm.exe           00000000037DE3C2  shr_strdata_init_         347  dshr_strdata_mod.F90
cesm.exe           00000000037282BB  init                      115  ch4FInundatedStreamType.F90
cesm.exe           00000000030ACF25  init                      241  ch4Mod.F90
cesm.exe           0000000000936BBB  clm_instinit              396  clm_instMod.F90
cesm.exe           00000000009290FC  initialize2               419  clm_initializeMod.F90
cesm.exe           000000000085A034  initializerealize         677  lnd_comp_nuopc.F90

Originally posted by @ekluzek in #3796 (comment)

[ekluzek]

OK, in looking into this more I did find a workaround.

For whatever reason this fails when

ch4finundatedmapalgo is the default value of "bilinear". But, when I use "nn", "consf", or "consd" it works!

Since. this is basically for single point simulations -- it seems pretty reasonable to use "nn". Although bilinear is still working for a lot of the mapalgo for single point cases.

[ekluzek]

I was able to show that all the case work if I change

ch4finundatedmapalgo = 'nn'

so this seems like the simple way forward.

[ekluzek]

The fail happens in ESMF on this call to ESMF_FieldRegridStore:

          if (trim(sdat%stream(ns)%mapalgo) == "bilinear") then
             call ESMF_FieldRegridStore(sdat%pstrm(ns)%field_stream, lfield_dst, &
                  routehandle=sdat%pstrm(ns)%routehandle, &
                  regridmethod=ESMF_REGRIDMETHOD_BILINEAR,  &
                  polemethod=ESMF_POLEMETHOD_ALLAVG, &
                  extrapMethod=ESMF_EXTRAPMETHOD_NEAREST_STOD, &
                  dstMaskValues=(/sdat%stream(ns)%dst_mask_val/), &
                  srcMaskValues=(/sdat%stream(ns)%src_mask_val/), &
                  srcTermProcessing=srcTermProcessing_Value, ignoreDegenerate=.true., rc=rc)

I turned ESMF_PET files on for this test:

SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.derecho_intel.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO

and didn't see anything interesting there.

The land log ends with the following...

lnd.log:

 Setting initial data to non-spun up values for CH4 Mod
 Reading ch4finundated namelist from file: lnd_in
  
 ch4finundated stream settings:
   stream_fldFileName_ch4finundated = 
 /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/paramdata/finundated_inversion
 data_0.9x1.25_c170706.nc
   stream_meshfile_ch4finundated    = 
 /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/paramdata/finundated_inversion
 data_0.9x1_ESMFmesh_cdf5_130621.nc
   ch4finundatedmapalgo             = bilinear
   stream_varnames                  = FWS_TWS_A       FWS_TWS_B       
(shr_strdata_init_from_inline) inline call for stream ch4 finundation
 
(shr_strdata_get_stream_nlev) Stream: 1 stream_nlev = 1
(shr_sdat_init) Creating field bundle array on model mesh for (lb,ub) of input data for stream 1
(shr_sdat_init)    adding field FWS_TWS_A to field bundle array 
(shr_sdat_init)    adding field FWS_TWS_B to field bundle array

[ekluzek]

Oh, it fails for the five tests at the top, but passes for these 9.

ERS_Ld1600_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.derecho_intel.clm-cropMonthlyNoinitial
ERS_Ld1640_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCrop.derecho_intel.clm-ciso_monthly_matrixcn_spinup
ERS_Ly20_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.derecho_intel.clm-cropMonthlyNoinitial--clm-matrixcnOn
ERS_Ly20_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.derecho_intel.clm-cropMonthlyNoinitial
ERS_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropQianRs.derecho_intel.clm-cropMonthOutput--clm-matrixcnOn_ignore_warnings
ERS_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropQianRs.derecho_intel.clm-cropMonthOutput
SMS_Ly1_Mmpi-serial.1x1_brazil.IHistClm60BgcQianRs.derecho_intel.clm-output_bgc_highfreq
SMS_Ly5_Mmpi-serial.1x1_brazil.IHistClm50BgcQianRs.derecho_intel.clm-newton_krylov_spinup
SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm60BgcCropQianRs.derecho_intel.clm-ciso_monthly

The gnu tests also all PASS and the izumi_nag tests PASS as well.

Oh, and there are two tests I didn't include in above:

SUBSETDATAPOINT_Ld5_D_Mmpi-serial.CLM_USRDAT.I2000Clm60BgcCropCrujra.derecho_intel.clm-default
SUBSETDATAREGION_Ld5_D_Mmpi-serial.CLM_USRDAT.I2000Clm60BgcCropCrujra.derecho_intel.clm-default

that also fail.

There are also tests that fail using the same point as other tests that PASS. And as such I wouldn't think it's a problem with the dataset for the points around the site being run on.