Add vertical advection

[brian-oneill]

Add the VertAdv module to Omega, providing data structures and methods for vertical advection. This module includes:

• Arrays for storing vertical velocity and tracer fluxes at layer interfaces.
• A method to compute the vertical velocity from the divergence of the horizontal velocity.
• Methods for computing the tendencies of thickness, horizontal velocity, and tracers due to vertical advection. The tracer tendency methods include multiple options for order of accuracy of the flux calculation, as well as algorithmic options for standard advection or flux-corrected transport (FCT).

Additional changes in this PR:

• Extend parallelForOuter with an optional argument to enable the use of scratch arrays in hierarchical parallelism.
• Add the auxiliary variable ProvThickness, which represents thickness after horizontal thickness flux. Further work required to fully implement provisional thickness.
• Remove BottomDepth from HorzMesh, this field was previously added to VertCoord.

This PR builds and passes the unit tests successfully on Chrysalis, pm-cpu, pm-gpu, and Frontier (CPU & GPU).

Checklist

• Documentation:
  • Design document has been generated and added to the docs
  • User's Guide has been updated
  • Developer's Guide has been updated
  • Documentation has been built locally and changes look as expected
• Linting
  • Pre-commit run locally
• Building
  • CMake build does not produce any new warnings from changes in this PR
• Testing
  • Add a comment to the PR titled Testing with the following:
    • Which machines CTest unit tests
      have been run on and indicate that are all passing.
    • The Polaris omega_pr test suite
      has passed, using the Polaris e3sm_submodules/Omega baseline
    • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
    • Indicate "All tests passed" or document failing tests
    • Document testing used to verify the changes including any tests that are added/modified/impacted.
  • New tests:
    • CTest unit tests for new features have been added per the approved design.
    • Polaris tests for new features have been added per the approved design (and included in a test suite)

Fixes #338

[mwarusz]

When running VERTADV_TEST with bounds checking enabled I get out-of-bounds errors. The reason is VertCoord::init(false) gets NVertLayers from OmegaMesh.nc and this value is used to create the member arrays of VertAdv in VertAdv::init(). The value of NVertLayers in VertAdv and MaxLayerCell get overwritten later in the convergence test, but the VertAdv member arrays (e.g. VertFlux) don't get resized.

[cbegeman]

Do we want to use

Omega/components/omega/src/ocn/HorzOperators.h

Line 139 in f2e951a

InterpCellToEdge(const HorzMesh *Mesh);

so that we can easily switch from anisotropic to isotropic interpolation for all variables (here and elsewhere)?

[brian-oneill]

We can’t use InterpCellToEdge here as is, since it takes an Array1DReal as input, while TotalVerticalVelocity is an Array2DReal. Supporting this would require either an overload or a templated version of the interpolator. It may make more sense to handle this outside this PR and identify any other places to implement this refactor.

[mark-petersen]

See compiled documentation here:

• user's guide, Vertical Advection
• developer's guide, Vertical Advection

[hyungyukang]

Using my time-stepping working branch (https://github.com/hyungyukang/Omega/tree/hkang/omega/merge-vert-vel
), I tested this PR with the merry-go-round test in Polaris. Since the test does not currently support Omega directly, I made several adjustments to run it manually. I will document the procedure for running the test manually in a separate comment later.

As expected, I obtained a convergence rate of 1.971 with VerticalTracerFluxOrder: 2, which matches the convergence rate obtained in MPAS-Ocean:

[hyungyukang]

Suggested change

	VAdv->computeTracerVAdvTend(TracerTend, TracerArray, ThicknessForVAdv,
	VAdv->computeTracerVAdvTend(LocTracerTend, TracerArray, ThicknessForVAdv,

[hyungyukang]

I’m approving this PR based on my testing and visual inspection. All the changes I requested have been addressed, and the code works as expected.

@brian-oneill, thank you very much for your hard work on this!

[mark-petersen]

Approving based on Hyun's convergence test above, and visual inspection. Successfully passed all cTests on perlmutter CPU and GPU with gnu compilers.

[cbegeman]

I ran the 5m merry-go-round case with Omega (above) and MPAS-Ocean (below) and the error patterns look very similar. The horizontal velocity for Omega is not yet available, as it relies on reconstruction to the x-direction.

[cbegeman]

Great work, @brian-oneill! I just wanted to check; has anyone yet verified the 3rd and 4th order convergence options? If not, I'll do this with the polaris merry-go-round test.

[hyungyukang]

Great work, @brian-oneill! I just wanted to check; has anyone yet verified the 3rd and 4th order convergence options? If not, I'll do this with the polaris merry-go-round test.

@cbegeman , I ran it last time with the 3rd convergence option. However, I was not able to obtain the 3rd order convergence rate for both Omega (1.981) and MPAS-Ocean (1.981). Could you please run it as well just in case I missed something?

[cbegeman]

Using one processor, I was able to reproduce @hyungyukang's results:

Testing whether this is fixed by #316

[cbegeman]

4th order advection with limiter (MPAS top, Omega bottom):

[cbegeman]

4th order advection without limiter (MPAS top, Omega bottom):

[hyungyukang]

4th order advection without limiter (MPAS top, Omega bottom)

@cbegeman , I ran the test on my end with 4th order advection without limiter and obtained the same convergence rate (1.980) for both models.

[mark-petersen]

There is currently a halo boundary error with the merry-go-round test, per a slack conversation with @cbegeman. Following rows are with 1, 4, and 8 CPUs. @brian-oneill confirmed that halos of normvel tendency and are initialized to correctly to 0. I suspect that the error at partition boundaries is due to the imposed horizontal velocity for this specific test.

[mark-petersen]

Update: Halo updates with RK2/RK4 fixed boundary error problems in previous post.

[cbegeman]

4th order vertical advection with limiter (MPAS-O top, Omega bottom):
without limiter here