CN Sparse Matrix multiply errors that show up with intel/2025.3.2 update

[ekluzek]

Updating to intel/2025.3.2 in ccs_config1.0.77 shows two new unexpected fails. The fails are an error in the SparseMatrixMultiply for CN matrix with errors that look like this:

cesm.log:

dec2454.hsn.de.hpc.ucar.edu 6:  ERROR in SparseMatrixMultiplyMod.F90 at line 973
dec2454.hsn.de.hpc.ucar.edu 62:  ERROR in SparseMatrixMultiplyMod.F90 at line 973
dec2454.hsn.de.hpc.ucar.edu 47:  Negative conc. in ch4tran. c,j,deficit (mol):           7           2
dec2454.hsn.de.hpc.ucar.edu 47:   1.046852068611785E-003
dec2454.hsn.de.hpc.ucar.edu 62: Image              PC                Routine            Line        Source
dec2454.hsn.de.hpc.ucar.edu 62: cesm.exe           0000000003A188E5  shr_abort_backtra         110  shr_abort_mod.F90
dec2454.hsn.de.hpc.ucar.edu 62: cesm.exe           0000000003A188A1  shr_abort_abort            65  shr_abort_mod.F90
dec2454.hsn.de.hpc.ucar.edu 62: cesm.exe           0000000003A196CE  shr_assert                 95  shr_assert_mod.F90.in
dec2454.hsn.de.hpc.ucar.edu 62: cesm.exe           0000000003A19A32  shr_assert_all_1d         112  shr_assert_mod.F90.in
dec2454.hsn.de.hpc.ucar.edu 62: cesm.exe           0000000002AFF8C8  spmp_b_acc                973  SparseMatrixMultiplyMod.F90
dec2454.hsn.de.hpc.ucar.edu 62: cesm.exe           0000000001256C95  cnsoilmatrix              614  CNSoilMatrixMod.F90
dec2454.hsn.de.hpc.ucar.edu 62: cesm.exe           000000000331EFB4  cndriverleaching         1113  CNDriverMod.F90
dec2454.hsn.de.hpc.ucar.edu 62: cesm.exe           00000000015BD670  ecosystemdynamics        1127  CNVegetationFacade.F90
dec2454.hsn.de.hpc.ucar.edu 62: cesm.exe           000000000091FCE9  clm_driver_mp_clm        1157  clm_driver.F90

For these tests:

ERP_D_P64x2_Ld3.f10_f10_mg37.I1850Clm50BgcCrop.derecho_intel.clm-default--clm-matrixcnOn_ignore_warnings        (RUN)
ERP_P64x2_D_Ld5.f10_f10_mg37.I1850Clm50Bgc.derecho_intel.clm-ciso--clm-matrixcnOn_ignore_warnings       (RUN)

Originally posted by @ekluzek in #3796 (comment)

[ekluzek]

Rerunning the second test got it to work (although it now shows a compare to baseline error). I've rerun the first one three times and am now doing a fourth attempt.

[ekluzek]

It worked on the 4th attempt. So this is a duplicate of the previous issue. So I'll close this one and leave that one open.