Verify global super cycling with RK4 is reasonable

[pwolfram]

Supercycling particle integration implemented in MPAS-Dev/MPAS#1484 should be verified for global runs.

[pwolfram]

cc @bradyrx @maltrud

[pwolfram]

@bradyrx, the runs you plan to do in the Southern Ocean would be perfect for this application.

What is the global time step you are currently using? I'm thinking you could easily try a 10X longer time frame for particles to start. Also, make sure you are using RK4:

config_AM_lagrPartTrack_timeIntegration = 4

The default is RK2 for efficiency when config_AM_lagrPartTrack_compute_interval = 'dt'

See #8.

[bradyrx]

@pwolfram,

I believe the global time step is default. Output from the current run shows :config_dt = "00:10:00" ;, assuming that's the proper config option you're referencing it is 10 minutes.

I'll make sure to use RK4 when implementing the particles. Should I set RK4 on these supercycling tests?

[pwolfram]

Yes, I'd recommend RK4 for the supercycling tests. It wouldn't hurt to also do RK2 for the highest-resolution time step too, just as a double check.

[bradyrx]

@pwolfram ,

I think similar to the sampling namelist entries, config_AM_lagrPartTrack_timeIntegration is not yet incorporated into E3SM as a whole, so the system doesn't recognize it when changed in user_nl_mpaso:

Calling /turquoise/usr/projects/climate/rileybrady/E3SM_supercycling/components/mpas-ocean/cime_config/buildnml
ERROR: Command: '/turquoise/usr/projects/climate/rileybrady/E3SM_supercycling/components/mpas-ocean/bld/build-namelist -infile /turquoise/usr/projects/climate/rileybrady/E3SM_supercycling/GMPAS-IAF.T62_oEC60to30v3.noSupercycling.RK4/Buildconf/mpasoconf/cesm_namelist -caseroot /turquoise/usr/projects/climate/rileybrady/E3SM_supercycling/GMPAS-IAF.T62_oEC60to30v3.noSupercycling.RK4 -casebuild /turquoise/usr/projects/climate/rileybrady/E3SM_supercycling/GMPAS-IAF.T62_oEC60to30v3.noSupercycling.RK4/Buildconf -cimeroot /turquoise/usr/projects/climate/rileybrady/E3SM_supercycling/cime -inst_string '' -decomp_prefix 'mpas-o.graph.info.' -date_stamp '161222' -ocn_grid 'oEC60to30v3' -ocn_forcing 'core_forced_restoring' -ocn_bgc 'no_bgc' -ocn_co2_type 'constant' -atm_co2_const_val '379.0' -ice_bgc 'no_ice_bgc' -ntasks_ocn '512' -ninst_ocn '1'' failed with error 'b"** ERROR: either config_am_lagrparttrack_timeintegration is not a valid MPASO namelist variable or config_am_lagrparttrack_timeintegration = '4' is not a valid value; please fix user_nl_mpaso. Note that config_am_lagrparttrack_timeintegration may appear in multiple namelists, in which case you need to specify the correct namelist in user_nl_mpaso using the format config_am_lagrparttrack_timeintegration&namelist_nml = '4', where &namelist_nml is the mpaso_in namelist containing config_am_lagrparttrack_timeintegration.**"' from dir '/turquoise/usr/projects/climate/rileybrady/E3SM_supercycling/GMPAS-IAF.T62_oEC60to30v3.noSupercycling.RK4/Buildconf/mpasoconf'

I'll go ahead and just set RK4 to default in the mpas Registry and set up the RK4 cases from there.

[pwolfram]

@bradyrx is doing testing with

• RK2 no supercycling
• RK4 no supercycling
• RK4 1 hr
• RK4 2 hr
• RK4 4 hr supercycling cases

[bradyrx]

@pwolfram @maltrud

Results from supercycling cases. This was a 60to30km g-case (GMPAS-IAF) physics-only simulation. Seeding was restricted to Southern Ocean and passive/surface float types (for some perspective on our use-case).

Case Description	Run Efficiency [SYPD]
No Supercycling, RK2	1.75
No Supercycling, RK4	1.36
1hr Supercycling, RK4	1.39
2hr Supercycling, RK4	1.42
4hr Supercycling, RK4	1.44

Per issue title, what further verification is required? This currently only gives compute efficiency for the various cases.

[pwolfram]

@bradyrx, this is strange. Does this suggest that running in "standard" mode with compute_dt, no super-cycling, is faster than any supercycled case with RK4? RK4 has about 2X the computational cost, so this makes sense.

I would suspect that No Supercycling, RK2 and compute_dt = 2*dt should be about the same, at least for standalone. Does this type of scaling make sense for compute?

I'm thinking this implies the runtime is dominated by the LIGHT IO. Is this the case for these runs (e.g., what is compute / IO)?

2hr vs 4hr supercycling with RK4 should be a speedup of about 2X for the compute bit of LIGHT. Is this what you are seeing in the log files? I'm not sure you really want to do advection at a longer time step than 4hrs.