Standard non-monotonic higher order horizontal advection copied from mpas-ocean to Omega. by overfelt · Pull Request #316 · E3SM-Project/Omega

overfelt · 2025-11-18T22:39:20Z

Checklist

overfelt · 2025-12-02T20:42:27Z

This is the basic standard higher order horizontal advection scheme. MPAS-Ocean also has a monotonic FCT transport scheme that includes modifications to this basic scheme which is not included with this PR.

Testing:
Unit tests for the new functions added to HorzOperators class.
(1) testsecondderivativeoncellDeterminePlanerPatchGeometry - Second derivative on a patch of elements test. This test creates a little patch of elements and defines the geometry from scratch by just hard coding a bunch of coordinates. Then the second derivative calculated by the HorzOperators class is compared to an analytic solution.
(2) testsecondderivativeoncellLeastSquaresFit - The LeastSquaresFit function is tested against an analytic solution.
(3) testsecondderivativeoncellDetermineSphericalPatchGeometry - Is like the testsecondderivativeoncellDeterminePlanerPatchGeometry unit test but on a spherical mesh.
(4) testsecondderivativeoncellconstructor - A trivial unit test that just checks that an instance of the class SecondDerivativeOnCell can be constructed.

philipwjones · 2025-12-05T16:35:44Z

@overfelt This is failing both the Horz Operators unit tests on Chrysalis and on Frontier (gpu only - cpu passes). The errors are memory errors - invalid pointer in free() on Chrysalis and inaccessible memory space (for array named XP) on Frontier gpu. So something isn't getting cleaned up correctly? I'll start reading through code to review...

overfelt · 2025-12-05T18:44:47Z

@philipwjones ,I only ran perlmutter gpu. I'll try setting up and running on Chrysalis gpu. Thanks.

mark-petersen · 2025-12-09T18:11:24Z

@overfelt, the tests also fail on perlmutter, both CPU and GPU, using the gnu compiler.

The following tests FAILED:
	 11 - HORZOPERATORS_PLANE_TEST (Failed)
	 12 - HORZOPERATORS_SPHERE_TEST (Failed)

Here are my instructions. Let me know if you can reproduce these errors. You will need to change to your own paths.

perlmutter CPU:

######### perlmutter CPU
#export CODEDIR=omega-develop
export CODEDIR=opr
export RUNDIR=test_omega_cpu

cd /global/homes/m/mpeterse/repos/E3SM/${CODEDIR}
git submodule update --init --recursive components/omega/external/GSW-C components/omega/external/yaml-cpp externals/ekat externals/scorpio cime externals/cpptrace
cd components/omega/

module load cmake
mkdir ${PSCRATCH}/runs/$RUNDIR
cd ${PSCRATCH}/runs/$RUNDIR

rm -rf build
mkdir build
cd build

# compiler options are:
export compiler=gnu
#export compiler=nvidia # not working 250421

export PARMETIS_ROOT=/global/cfs/cdirs/e3sm/software/polaris/pm-cpu/spack/dev_polaris_0_9_0_${compiler}_mpich/var/spack/environments/dev_polaris_0_9_0_${compiler}_mpich/.spack-env/view

# nvidia or gnu compiler:
cmake \
   -DOMEGA_CIME_COMPILER=${compiler} \
   -DOMEGA_BUILD_TYPE=Release \
   -DOMEGA_CIME_MACHINE=pm-cpu \
   -DOMEGA_PARMETIS_ROOT=${PARMETIS_ROOT}\
   -DOMEGA_BUILD_TEST=ON \
   -DOMEGA_VECTOR_LENGTH=1 \
   -Wno-dev \
   -S /global/homes/m/mpeterse/repos/E3SM/${CODEDIR}/components/omega -B .
# note OMEGA_VECTOR_LENGTH=8 fails MPI tests on CPUs.
cd ${PSCRATCH}/runs/$RUNDIR/build
./omega_build.sh

# linking:
cd test
ln -isf /global/homes/m/mpeterse/meshes/omega/O*nc .
cp /global/homes/m/mpeterse/repos/E3SM/${CODEDIR}/components/omega/configs/Default.yml omega.yml

# run test:
salloc --nodes 1 --qos interactive --time 01:00:00 --constraint cpu --account=m4572 # or e3sm
cd ${PSCRATCH}/runs/${RUNDIR}/build

./omega_ctest.sh

perlmutter GPU:

######### perlmutter GPU
salloc --nodes 4 --qos interactive --time 01:00:00 --constraint gpu --tasks-per-node=2 --gpus-per-task 1 --account=m4572_g # or e3sm_g

# perlmutter has nodes with either 40 or 80 gb of high bandwidth memory, and the system defaults to 40. You can ask for 80 gb nodes with the sbatch flag --constraint="gpu&hbm80gb"

export CODEDIR=opr
#export CODEDIR=omega-develop
export RUNDIR=test_omega_gpu
mkdir ${PSCRATCH}/runs/$RUNDIR
cd !$

rm -rf build
mkdir build
cd build
module load cmake

# compiler options are:
export compiler=gnugpu
#export compiler=nvidiagpu

export PARMETIS_ROOT=/global/cfs/cdirs/e3sm/software/polaris/pm-gpu/spack/dev_polaris_0_9_0_${compiler}_mpich/var/spack/environments/dev_polaris_0_9_0_${compiler}_mpich/.spack-env/view
cmake \
   -DOMEGA_CIME_COMPILER=${compiler} \
   -DOMEGA_BUILD_TYPE=Release \
   -DOMEGA_CIME_MACHINE=pm-gpu \
   -DOMEGA_PARMETIS_ROOT=${PARMETIS_ROOT}\
   -DOMEGA_BUILD_TEST=ON \
   -Wno-dev \
   -DOMEGA_MPI_ON_DEVICE:BOOL=OFF \
   -S /global/homes/m/mpeterse/repos/E3SM/${CODEDIR}/components/omega -B .
# needed for compiler bug: OMEGA_MPI_ON_DEVICE:BOOL=OFF. See https://github.com/E3SM-Project/Omega/issues/214
./omega_build.sh

# linking:
cd test
ln -isf /global/homes/m/mpeterse/meshes/omega/O*nc .
cp /global/homes/m/mpeterse/repos/E3SM/${CODEDIR}/components/omega/configs/Default.yml omega.yml

cd ..
./omega_ctest.sh

philipwjones · 2025-12-17T17:44:36Z

With the recent commit, this now passes CTests on Chrysalis and Frontier (cpu/gpu). Thanks @overfelt

mark-petersen · 2025-12-23T20:33:51Z

This passes CPU and GPU tests on perlmutter. Due to the merge of #314, this now has some conflicts to be resolved. Please rebase on the current head, and we will proceed with the review after the break. Thanks!

philipwjones

I have not done an exhaustive comparison to the paper or MPAS to verify the terms, but I did browse the code somewhat and this appears to be working fine based on convergence testing and passes all unit tests. Some of the code (esp HorzUtil) does not match Omega coding style, but I won't hold this up. At some point in the near future, I think we need to break up the monolithic tendency file.

brian-oneill · 2026-01-12T15:33:56Z

I ran the ctests successfully on pm-cpu, pm-gpu, and Chrysalis.

I agree with Phil, I think this is fine to move forward. Just FYI on Omega coding style, we're using PascalCase for variable names and camelCase for function names. This PR is just the standard horizontal advection algorithm so it's fine as is, but I think the monotonic advection algortihm is complex enough to warrant moving horizontal advection to it's own file, possibly as a separate module.

cbegeman · 2026-01-27T21:17:12Z

@overfelt Glad to see this coming along! Let me know when this is ready to run Polaris tests and if you need any help getting those going

overfelt · 2026-01-27T21:24:00Z

What Polaris tests should I be running? I have only run the spherical cosine_bell test in my development.

cbegeman · 2026-01-27T23:11:43Z

What Polaris tests should I be running? I have only run the spherical cosine_bell test in my development.

It would be
polaris suite -c ocean -t sphere_transport_with_viz ...

That includes the icos meshes. After those pass, then you can move on to the qu meshes (just make substitutions in the test case name on setup).

These convergence tests include some higher-resolutions meshes that have been hitting IO errors. If you want to bypass those, use the latest omega_pr suite modified just recently here E3SM-Project/polaris#455.

The order of the advection scheme tested will be whatever the default option is in Omega or you'll have to manually change it (until we update Polaris). If that's too much of a pain, let me know and I can automate it in a polaris branch.

And then you can just visually examine the convergence plot and the state visualizations since the analysis PASS/FAIL will be based off of hard-coded orders of convergence (https://github.com/E3SM-Project/polaris/blob/bd87bf281986b9a28ad63370d0ad495140f4d84d/polaris/tasks/ocean/sphere_transport/analysis.py#L76-L79)

components/omega/src/ocn/Tendencies.cpp

mark-petersen · 2026-02-20T18:52:11Z

These tests fail with the current head on perlmutter CPU and GPU with gnu:

The following tests FAILED:
	 11 - HORZOPERATORS_PLANE_TEST (Failed)
	 12 - HORZOPERATORS_SPHERE_TEST (Failed)

These are my commands

######### perlmutter CPU
export CODEDIR=opr
export CODEPATH=${HOME}/repos/E3SM/${CODEDIR}
export RUNDIR=test_omega_cpu

cd $CODEPATH
git submodule update --init --recursive components/omega/external/GSW-C components/omega/external/yaml-cpp externals/ekat externals/scorpio cime externals/cpptrace
cd components/omega/

module load cmake
mkdir ${PSCRATCH}/runs/$RUNDIR
cd ${PSCRATCH}/runs/$RUNDIR

rm -rf build
mkdir build
cd build

# compiler options are:
export compiler=gnu
#export compiler=nvidia # not working 250421

export PARMETIS_ROOT=/global/cfs/cdirs/e3sm/software/polaris/pm-cpu/spack/dev_polaris_0_9_0_${compiler}_mpich/var/spack/environments/dev_polaris_0_9_0_${compiler}_mpich/.spack-env/view

# nvidia or gnu compiler:
cmake \
   -DOMEGA_CIME_COMPILER=${compiler} \
   -DOMEGA_BUILD_TYPE=Release \
   -DOMEGA_CIME_MACHINE=pm-cpu \
   -DOMEGA_PARMETIS_ROOT=${PARMETIS_ROOT}\
   -DOMEGA_BUILD_TEST=ON \
   -DOMEGA_VECTOR_LENGTH=1 \
   -Wno-dev \
   -S ${CODEPATH}/components/omega -B .
# note OMEGA_VECTOR_LENGTH=8 fails MPI tests on CPUs.
cd ${PSCRATCH}/runs/$RUNDIR/build
./omega_build.sh

# linking:
cd test
ln -isf /global/homes/m/mpeterse/meshes/omega/O*nc .
cp ${CODEPATH}/components/omega/configs/Default.yml omega.yml

# run test:
salloc --nodes 1 --qos interactive --time 01:00:00 --constraint cpu --account=m4572 # or e3sm
cd ${PSCRATCH}/runs/${RUNDIR}/build

./omega_ctest.sh

######### perlmutter GPU
salloc --nodes 4 --qos interactive --time 01:00:00 --constraint gpu --tasks-per-node=2 --gpus-per-task 1 --account=m4572_g # or e3sm_g

# perlmutter has nodes with either 40 or 80 gb of high bandwidth memory, and the system defaults to 40. You can ask for 80 gb nodes with the sbatch flag --constraint="gpu&hbm80gb"

export CODEDIR=opr
export CODEPATH=${HOME}/repos/E3SM/${CODEDIR}
export RUNDIR=test_omega_gpu
mkdir ${PSCRATCH}/runs/$RUNDIR
cd !$

rm -rf build
mkdir build
cd build
module load cmake

# compiler options are:
export compiler=gnugpu
#export compiler=nvidiagpu

export PARMETIS_ROOT=/global/cfs/cdirs/e3sm/software/polaris/pm-gpu/spack/dev_polaris_0_9_0_${compiler}_mpich/var/spack/environments/dev_polaris_0_9_0_${compiler}_mpich/.spack-env/view
cmake \
   -DOMEGA_CIME_COMPILER=${compiler} \
   -DOMEGA_BUILD_TYPE=Release \
   -DOMEGA_CIME_MACHINE=pm-gpu \
   -DOMEGA_PARMETIS_ROOT=${PARMETIS_ROOT}\
   -DOMEGA_BUILD_TEST=ON \
   -Wno-dev \
   -DOMEGA_MPI_ON_DEVICE:BOOL=OFF \
   -S ${CODEPATH}/components/omega -B .
# needed for compiler bug: OMEGA_MPI_ON_DEVICE:BOOL=OFF. See https://github.com/E3SM-Project/Omega/issues/214
./omega_build.sh

# linking:
cd test
ln -isf /global/homes/m/mpeterse/meshes/omega/O*nc .
cp ${CODEPATH}/components/omega/configs/Default.yml omega.yml

cd ..
./omega_ctest.sh

mwarusz · 2026-02-20T19:19:34Z

@overfelt Apologies. This change in my conservation fixes is what broke your tests:

+      const Real sphereRadius = REarth;
-      const Real sphereRadius = distance(XCell(0), YCell(0), ZCell(0));

I haven't looked yet to check if there is an easy way to fix them.

xylar · 2026-02-20T19:36:26Z

We do not want to be modifying the Earth radius or the coordinates on Omega. We need them to be fixed on the Polaris side.

overfelt · 2026-02-20T22:10:53Z

@mwarusz Your fix is the correct thing to do and the fix was trivial. Just had to copy your fix to the unit test so the check passes.

cbegeman · 2026-02-23T16:23:21Z

components/omega/configs/Default.yml

    BottomDragTendencyEnable: false
    BottomDragCoeff: 0.0
    TracerHorzAdvTendencyEnable: true
+    TracerHorzAdvTendencyOrder: 1


I think this config option belongs in the Advection section, and we should reserve the Tendencies section for options that are related to turning tendencies on/off. I know this hasn't been the case so far, so if folks have a different vision for organization, let me know.

Suggested change

TracerHorzAdvTendencyOrder: 1

HorzTracerFluxOrder: 1

In #328, the config option is VerticalTracerFluxOrder, so this change would make it consistent.

…uxOrder for consistentsy.

…nventions.

…ames Changes to match MPAS-O convergence rates in tracer advection tests

overfelt marked this pull request as draft November 18, 2025 23:00

overfelt marked this pull request as ready for review November 19, 2025 13:18

vanroekel requested review from brian-oneill, mark-petersen and philipwjones December 1, 2025 14:08

overfelt force-pushed the overfelt/HigherOrderTendency_Cleanup branch from 5ae40c2 to 1055c8f Compare December 2, 2025 20:21

overfelt closed this Dec 2, 2025

overfelt reopened this Dec 2, 2025

overfelt force-pushed the overfelt/HigherOrderTendency_Cleanup branch from 35bf9b3 to 803b4a7 Compare December 9, 2025 20:38

overfelt force-pushed the overfelt/HigherOrderTendency_Cleanup branch from 9ec4b1a to 5eb5850 Compare January 3, 2026 15:37

philipwjones approved these changes Jan 8, 2026

View reviewed changes

overfelt requested a review from mwarusz February 6, 2026 15:01

hyungyukang reviewed Feb 11, 2026

View reviewed changes

components/omega/src/ocn/Tendencies.cpp Outdated Show resolved Hide resolved

cbegeman reviewed Feb 23, 2026

View reviewed changes

cbegeman mentioned this pull request Feb 23, 2026

Add vertical advection #328

Open

19 tasks

overfelt force-pushed the overfelt/HigherOrderTendency_Cleanup branch from 653f4e4 to 2da05b0 Compare February 24, 2026 19:52

overfelt and others added 21 commits February 25, 2026 11:46

Rename flag used to trigger higher order advection.

b73ed2e

Clang format.

2da7e1d

Clang format with newer version.

9bd0099

Add documentation for the higher order horizontal flux calculation.

84caf50

Remove whitespace the linter disapproves of.

de4fcf1

Fix off-by-one array size in unit test.

01659de

Fix a couple of signed/unsigned int comparison compiler warnings.

58781e7

Fix Kokkos View host/device incompatibility errors.

24358e6

Fix up use of VCoord after rebase from develop.

530f89f

Clang format

1312bac

Clang Format.

f4081c2

Fix compiler errors after merge from develop.

c9870a5

Forgot to zero out tendecies before summing into values.

afe0652

Fix for parallel: NCellsOwned -> NCellsAll/

27517c9

Fix conservation issue in high order advection

565206d

Clang format.

ad8f892

Fix for unit test. Use actual earth radius: REarth.

a776bc0

Fix setting of TracerHorzAdvTendencyEnable flag based on code review.

8b39eb8

Clang format.

8f19d72

Change parameter name from TracerHorzAdvTendencyOrder to HorzTracerFl…

9c1830b

…uxOrder for consistentsy.

Changing the variable names to PascalCase to follow Omega’s naming co…

06fab60

…nventions.

overfelt force-pushed the overfelt/HigherOrderTendency_Cleanup branch from 487843b to 06fab60 Compare February 25, 2026 18:58

mwarusz and others added 8 commits February 25, 2026 14:32

Add ForceLowOrder option to high-order tracer advection class

6a361f2

Use high-order tracer adv class in tendency terms test

faba875

Only use high-order tracer adv in tendencies

4b869b7

Rip out old tracer advection

8f99187

Rename: TracerHighOrderHorzAdv -> TracerHorzAdv

f75d949

Move some options to the advection section

595bca4

Fix size of HighOrderFlxHorz

a5bd1cf

Merge pull request #2 from mwarusz/mwarusz/omega/high-order-adv-for-j…

39e88c7

…ames Changes to match MPAS-O convergence rates in tracer advection tests

Conversation

overfelt commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

overfelt commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philipwjones commented Dec 5, 2025

Uh oh!

overfelt commented Dec 5, 2025

Uh oh!

mark-petersen commented Dec 9, 2025

Uh oh!

philipwjones commented Dec 17, 2025

Uh oh!

mark-petersen commented Dec 23, 2025

Uh oh!

philipwjones left a comment

Choose a reason for hiding this comment

Uh oh!

brian-oneill commented Jan 12, 2026

Uh oh!

cbegeman commented Jan 27, 2026

Uh oh!

overfelt commented Jan 27, 2026 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbegeman commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mark-petersen commented Feb 20, 2026

Uh oh!

mwarusz commented Feb 20, 2026

Uh oh!

xylar commented Feb 20, 2026

Uh oh!

overfelt commented Feb 20, 2026

Uh oh!

cbegeman Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

cbegeman Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

overfelt commented Nov 18, 2025 •

edited

Loading

overfelt commented Dec 2, 2025 •

edited

Loading

overfelt commented Jan 27, 2026 via email •

edited

Loading

cbegeman commented Jan 27, 2026 •

edited

Loading