Exchange tracer halos at every RK stage

[mwarusz]

Checklist

• Documentation:
  • Design document has been generated and added to the docs
  • User's Guide has been updated
  • Developer's Guide has been updated
  • Documentation has been built locally and changes look as expected
• Linting
  • Pre-commit run locally
• Building
  • CMake build does not produce any new warnings from changes in this PR
• Testing
  • Add a comment to the PR titled Testing with the following:
    • Which machines CTest unit tests
      have been run on and indicate that are all passing.
    • The Polaris omega_pr test suite
      has passed, using the Polaris e3sm_submodules/Omega baseline
    • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
    • Indicate "All tests passed" or document failing tests
    • Document testing used to verify the changes including any tests that are added/modified/impacted.
    • Performance related PRs: Please include a relevant PACE experiment link documenting performance before and after.
  • New tests:
    • CTest unit tests for new features have been added per the approved design.
    • Polaris tests for new features have been added per the approved design (and included in a test suite)
• Stealth Features
  • If any stealth features are included in the PR, please confirm that they have been documented.

[mark-petersen]

This is required for high order horizontal advection on multi-processor. It also was required for the merry-go-round test for vertical advection. See before error versus after with this fix.

[cbegeman]

Approving on the basis of testing with merry-go-round case and vertical advection.

[brian-oneill]

Approving based on inspection and successful execution of ctests on pm-cpu and pm-gpu, as well as above mentioned tests.