Running COMPASS in the performance_test step takes a long time without producing any output

[gufguan-nature]

I want to add a new ocean/sea ice regionally refined mesh (RRM) and I successfully pass the Running the mesh test case:

ocean/global_ocean/YAM10to60/mesh

• step: base_mesh
• step: remap_topography
• step: cull_mesh
  test execution: SUCCESS
  test runtime: 11:31
  Test Runtimes:
  11:31 PASS ocean_global_ocean_YAM10to60_mesh
  Total runtime 11:32
  PASS: All passed successfully!

I am following the tutorial to run “Adding an initial condition and performance test.” The initial_state step runs without any problem, but the performance_test step takes a very long time and produces no output.

case_outputs/ocean_global_ocean_YAM10to60_WOA23_performance_test.log：

compass calling: compass.ocean.tests.global_ocean.performance_test.PerformanceTest.run()
inherited from: compass.testcase.TestCase.run()
in /share/home/dq147/Gufguan/software/add-yet-another-mesh/compass/testcase.py

compass calling: compass.run.serial._run_test()
in /share/home/dq147/Gufguan/software/add-yet-another-mesh/compass/run/serial.py

Running steps:
forward

• step: forward

compass calling: compass.ocean.tests.global_ocean.forward.ForwardStep.runtime_setup()
in /share/home/dq147/Gufguan/software/add-yet-another-mesh/compass/ocean/tests/global_ocean/forward.py

Warning: replacing namelist options in namelist.ocean
config_dt = 00:05:00
config_btr_dt = 00:00:15

compass calling: compass.ocean.tests.global_ocean.forward.ForwardStep.run()
in /share/home/dq147/Gufguan/software/add-yet-another-mesh/compass/ocean/tests/global_ocean/forward.py

Warning: replacing namelist options in namelist.ocean
config_pio_num_iotasks = 6
config_pio_stride = 48
Running: gpmetis graph.info 288

---

METIS 5.0 Copyright 1998-13, Regents of the University of Minnesota
(HEAD: , Built on: Oct 4 2024, 17:55:59)
size of idx_t: 32bits, real_t: 32bits, idx_t *: 64bits

Graph Information -----------------------------------------------------------
Name: graph.info, #Vertices: 235213, #Edges: 697445, #Parts: 288

Options ---------------------------------------------------------------------
ptype=kway, objtype=cut, ctype=shem, rtype=greedy, iptype=metisrb
dbglvl=0, ufactor=1.030, no2hop=NO, minconn=NO, contig=NO, nooutput=NO
seed=-1, niter=10, ncuts=1

Direct k-way Partitioning ---------------------------------------------------

• Edgecut: 27962, communication volume: 28773.

• Balance:
  constraint #0: 1.030 out of 0.001

• Most overweight partition:
  pid: 244, actual: 841, desired: 816, ratio: 1.03.

• Subdomain connectivity: max: 9, min: 1, avg: 5.21

• There are 4 non-contiguous partitions.
  Total components after removing the cut edges: 292,
  max components: 2 for pid: 30.

Timing Information ----------------------------------------------------------
I/O: 0.053 sec
Partitioning: 0.210 sec (METIS time)
Reporting: 0.032 sec

Memory Information ----------------------------------------------------------
Max memory used: 22.863 MB

---

Running: mpirun -np 288 ./ocean_model -n namelist.ocean -s streams.ocean
compass calling: compass.ocean.tests.global_ocean.performance_test.PerformanceTest.run()
inherited from: compass.testcase.TestCase.run()
in /share/home/dq147/Gufguan/software/add-yet-another-mesh/compass/testcase.py

compass calling: compass.run.serial._run_test()
in /share/home/dq147/Gufguan/software/add-yet-another-mesh/compass/run/serial.py

Running steps:
forward

• step: forward

compass calling: compass.ocean.tests.global_ocean.forward.ForwardStep.runtime_setup()
in /share/home/dq147/Gufguan/software/add-yet-another-mesh/compass/ocean/tests/global_ocean/forward.py

Warning: replacing namelist options in namelist.ocean
config_dt = 00:05:00
config_btr_dt = 00:00:15

compass calling: compass.ocean.tests.global_ocean.forward.ForwardStep.run()
in /share/home/dq147/Gufguan/software/add-yet-another-mesh/compass/ocean/tests/global_ocean/forward.py

Warning: replacing namelist options in namelist.ocean
config_pio_num_iotasks = 6
config_pio_stride = 48
Running: gpmetis graph.info 288

---

METIS 5.0 Copyright 1998-13, Regents of the University of Minnesota
(HEAD: , Built on: Oct 4 2024, 17:55:59)
size of idx_t: 32bits, real_t: 32bits, idx_t *: 64bits

Graph Information -----------------------------------------------------------
Name: graph.info, #Vertices: 235213, #Edges: 697445, #Parts: 288

Options ---------------------------------------------------------------------
ptype=kway, objtype=cut, ctype=shem, rtype=greedy, iptype=metisrb
dbglvl=0, ufactor=1.030, no2hop=NO, minconn=NO, contig=NO, nooutput=NO
seed=-1, niter=10, ncuts=1

Direct k-way Partitioning ---------------------------------------------------

• Edgecut: 27962, communication volume: 28773.

• Balance:
  constraint #0: 1.030 out of 0.001

• Most overweight partition:
  pid: 244, actual: 841, desired: 816, ratio: 1.03.

• Subdomain connectivity: max: 9, min: 1, avg: 5.21

• There are 4 non-contiguous partitions.
  Total components after removing the cut edges: 292,
  max components: 2 for pid: 30.

Timing Information ----------------------------------------------------------
I/O: 0.060 sec
Partitioning: 0.213 sec (METIS time)
Reporting: 0.033 sec

Memory Information ----------------------------------------------------------
Max memory used: 22.863 MB

---

Running: mpirun -np 288 ./ocean_model -n namelist.ocean -s streams.ocean

compass/ocean/tests/global_ocean/mesh/yam10to60/namelist.split_explicit_ab2 is:

config_time_integrator = 'split_explicit_ab2'
config_run_duration = '0000_01:00:00'
config_use_mom_del2 = .true.
config_mom_del2 = 10.0
config_use_mom_del4 = .true.
config_mom_del4 = 1.5e10
config_hmix_scaleWithMesh = .true.
config_use_GM = .true.
config_GM_closure = 'constant'
config_GM_constant_kappa = 600.0

compass/ocean/tests/global_ocean/mesh/yam10to60/yam10to60.cfg is:

Options related to the vertical grid

[vertical_grid]

the type of vertical grid

grid_type = index_tanh_dz

Number of vertical levels

vert_levels = 64

Depth of the bottom of the ocean

bottom_depth = 5500.0

The minimum layer thickness

min_layer_thickness = 10.0

The maximum layer thickness

max_layer_thickness = 250.0

The characteristic number of levels over which the transition between

the min and max occurs

transition_levels = 28

options for global ocean testcases

[global_ocean]

the approximate number of cells in the mesh

approx_cell_count = 240000

metadata related to the mesh

the prefix (e.g. QU, EC, WC, SO)

prefix = YAM

a description of the mesh and initial condition

mesh_description = Yet Another Mesh (YAM) is regionally refined around the Amazon
River delta. It is used in E3SM version ${e3sm_version} for
studies of blah, blah. It has Enhanced resolution (${min_res} km)
around the Amazon delta, 20-km resolution in the northern South
Atlantic, 35-km resolution at the poles, 60-km at mid
latitudes, and 30-km at the equator. This mesh has <<>>
vertical levels.

E3SM version that the mesh is intended for

e3sm_version = 3

The revision number of the mesh, which should be incremented each time the

mesh is revised

mesh_revision = 1

the minimum (finest) resolution in the mesh

min_res = 10

the maximum (coarsest) resolution in the mesh, can be the same as min_res

max_res = 60

The URL of the pull request documenting the creation of the mesh

pull_request = https://github.com/MPAS-Dev/compass/pull/XXX

config options related to initial condition and diagnostics support files

for E3SM

[files_for_e3sm]

CMIP6 grid resolution

cmip6_grid_res = 180x360

[gufguan-nature]

log.txt

this is /share/home/dq147/Gufguan/model/compass_tests/tests_20230527/yam10to60_ec/ocean/global_ocean/YAM10to60/WOA23/performance_test/forward/log.ocean.0000.out

[gufguan-nature]

namelist_ocean.txt

this is /share/home/dq147/Gufguan/model/compass_tests/tests_20230527/yam10to60_ec/ocean/global_ocean/YAM10to60/WOA23/performance_test/forward/namelist.ocean

[xylar]

@gufguan-nature, are you able to run smaller MPAS-Ocean compass tests? The performance test is the first time in the "global ocean" workflow that MPAS-Ocean actually gets run in forward mode. It seems possible that your build of MPAS-Ocean is broken in general, though it seems you must have been able to run in "init mode" to create an ocean initial condition.

Please run one or more of the tests in the pr test suite to see if you can at least perform small, short forward runs with MPAS-Ocean.

Beyond that, unfortunately, none of the information you provided indicates anything obviously problematic in your setup.

[gufguan-nature]

   你的邮件已收到，谢谢！

[gufg254]

I am @gufguan-nature

when I set the YAM10to60 case like

    dlon = 0.1
    dlat = dlon
    nlon = int(360. / dlon) + 1
    nlat = int(180. / dlat) + 1
    lon = np.linspace(-180., 180., nlon)
    lat = np.linspace(-90., 90., nlat)
    cell_width = 60.0 * np.ones((nlat, nlon))

I can pass the Running the mesh test case, Adding an initial condition and performance test, Adding a dynamic adjustment test and Adding a files for E3SM test.

ocean/global_ocean/YAM10to60/WOA23/init

• step: initial_state
  test execution: SUCCESS
  test runtime: 01:14
  ocean/global_ocean/YAM10to60/WOA23/performance_test
• step: forward
  test execution: SUCCESS
  test runtime: 00:34
  Test Runtimes:
  01:14 PASS ocean_global_ocean_YAM10to60_WOA23_init
  00:34 PASS ocean_global_ocean_YAM10to60_WOA23_performance_test
  Total runtime 01:48
  PASS: All passed successfully!

However, when I Switching to an EC30to60 base resolution:

    lat = np.linspace(-90., 90., nlat)
    cell_width_vs_lat = mdt.EC_CellWidthVsLat(lat)
    cell_width = np.outer(cell_width_vs_lat, np.ones([1, lon.size]))

I can pass the Running the mesh test case and initial condition but stop in performance test
ocean/global_ocean/YAM10to60_ec/WOA23/init

• step: initial_state
  test execution: SUCCESS
  test runtime: 01:20
  ocean/global_ocean/YAM10to60_ec/WOA23/performance_test
• step: forward

It keeps running with no output and no errors reported

[gufg254]

@xylar

[xylar]

@gufg254, in the forward step's subdirectory:

ocean/global_ocean/Icos/YAM10to60_ec/performance_test/forward

you will see various files including at least log.ocean.0000.out and possibly one or more log.ocean.XXX.err files. Could you share the contents of those?

[gufg254]

@xylar I don't have thelog.ocean.XXX.errfiles.

[dq147@mgt01 forward]$ ll
total 2202
drwxr-xr-x 2 dq147 user01    4096 Dec 18 09:38 analysis_members
lrwxrwxrwx 1 dq147 user01     152 Dec 18 09:36 forcing_data.nc -> /share/home/dq147/Gufguan/model/compass_tests/tests_2025/yam10to60_ec/ocean/global_ocean/YAM10to60_ec/WOA23/init/initial_state/init_mode_forcing_data.nc
lrwxrwxrwx 1 dq147 user01     137 Dec 18 09:36 graph.info -> /share/home/dq147/Gufguan/model/compass_tests/tests_2025/yam10to60_ec/ocean/global_ocean/YAM10to60_ec/WOA23/init/initial_state/graph.info
-rw-r--r-- 1 dq147 user01  850132 Dec 18 09:38 graph.info.part.288
lrwxrwxrwx 1 dq147 user01     143 Dec 18 09:36 init.nc -> /share/home/dq147/Gufguan/model/compass_tests/tests_2025/yam10to60_ec/ocean/global_ocean/YAM10to60_ec/WOA23/init/initial_state/initial_state.nc
-rw-r--r-- 1 dq147 user01     177 Dec 18 09:36 job_script.sh
lrwxrwxrwx 1 dq147 user01      71 Dec 18 09:36 load_compass_env.sh -> /share/home/dq147/add-yet-another-mesh-1/load_compass_yam_gun1_mpich.sh
-rw-r--r-- 1 dq147 user01   78094 Dec 18 09:38 log.ocean.0000.out
lrwxrwxrwx 1 dq147 user01     131 Dec 18 09:36 mesh.nc -> /share/home/dq147/Gufguan/model/compass_tests/tests_2025/yam10to60_ec/ocean/global_ocean/YAM10to60_ec/mesh/cull_mesh/culled_mesh.nc
-rw-r--r-- 1 dq147 user01   47054 Dec 18 09:38 namelist.ocean
lrwxrwxrwx 1 dq147 user01      87 Dec 18 09:36 ocean_model -> /share/home/dq147/add-yet-another-mesh-1/E3SM-Project/components/mpas-ocean/ocean_model
lrwxrwxrwx 1 dq147 user01     145 Dec 18 09:36 performance_test.cfg -> /share/home/dq147/Gufguan/model/compass_tests/tests_2025/yam10to60_ec/ocean/global_ocean/YAM10to60_ec/WOA23/performance_test/performance_test.cfg
drwxr-xr-x 2 dq147 user01    4096 Dec 18 09:38 restarts
-rw-r--r-- 1 dq147 user01 1254487 Dec 18 09:36 step.pickle
-rw-r--r-- 1 dq147 user01    3072 Dec 18 09:36 streams.ocean
[dq147@mgt01 forward]$ pwd
/share/home/dq147/Gufguan/model/compass_tests/tests_2025/yam10to60_ec/ocean/global_ocean/YAM10to60_ec/WOA23/performance_test/forward
[dq147@mgt01 forward]$ 

[xylar]

Please share the contents of log.ocean.0000.out.

[xylar]

Also, here are some tricks for formatting text so it's more readable on GitHub:
https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax
I see you may be aware of them, since one of your messages used Markdown.

[gufg254]

log.ocean.0000.txt

[xylar]

Hmm, unfortunately that log doesn't tell me why the run is hanging either. You would have to debug MPAS-Ocean yourself by adding print statements or using a debugger. We won't really be much help there.

[xylar]

Sorry, I know I've had you running in circles. I now see you had shared the log file earlier as well. It's frustrating that there isn't more useful output, I know.