Hillslope test now failing because a file was in scratch, must have been scrubbed

[ekluzek]

Brief summary of bug

The HillclopeC test is now failing, because it looks like a scratch file pointed to was scrubbed:

SMS_D_Mmpi-serial_Ld5.5x5_amazon.I2000Clm60Bgc.derecho_gnu.clm-HillslopeC (RUN)

General bug information

CTSM version you are using: ctsm5.4.021-99-gbd6c29cb (add_mizuRoute branch off of b4b-dev)

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected:
Just the Hillslope testmods: Hillslope, HillslopC, HillslopeD, HillslopeFromFile
so the test-suites: aux_clm, hillslope and aux_clm_mpi_serial

Details of bug

The file doesn't exist so it now aborts at runtime with an error

check_input_data also fails with:

./check_input_data 
Loading input file list: 'Buildconf/mosart.input_data_list'
Loading input file list: 'Buildconf/datm.input_data_list'
Loading input file list: 'Buildconf/ctsm.input_data_list'
Model ctsm missing file hillslope_file = '/glade/derecho/scratch/samrabin/hillslopes_5x5_amazon/hand_analysis_global/combined/hilldata_5x5_amazon_hist_2000_78pfts_c240216.nc'
Loading input file list: 'Buildconf/cpl.input_data_list'

Important output or errors that show the problem

This file points to scratch:

git grep scratch
cime_config/testdefs/testmods_dirs/clm/Hillslope/shell_commands:    hillslope_file='/glade/derecho/scratch/samrabin/hillslopes_5x5_amazon/hand_analysis_global/combined/hilldata_5x5_amazon_hist_2000_78pfts_c240216.nc'

lnd.log has:

022.nc
 Opened existing file /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_esmf/ctsm5.4.0/surfdata_5x5_amazon_hist_2000_78pfts_c251022.nc         491
(GETFIL): attempting to find local file 
hilldata_5x5_amazon_hist_2000_78pfts_c240216.nc
(GETFIL): failed getting file from full path: /glade/derecho/scratch/samrabin/hillslopes_5x5_amazon/hand_analysis_global/combined/hilldata_5x5_amazon_hist_2000_78pfts_c240216.nc                                                                                                                                                                                                                                                                                                                                                                                             
 GETFIL: FAILED to get /glade/derecho/scratch/samrabin/hillslopes_5x5_amazon/hand_analysis_global/combined/hilldata_5x5_amazon_hist_2000_78pfts_c240216.nc

[ekluzek]

The underlying issue was already catalogued in #3477. Since, we test so often the file would've been touched every time we test which is often enough that it's surprising it happened. Although probably we weren't testing as much because of the LMWG meeting, and apparently that was long enough that it got scrubbed?

[samsrabin]

This is resolved in ctsm5.4.022 with an updated path. For now, I've made a softlink.