feat: Add regrid functionality based on match geometry

[eliascapriles-NOAA]

Implements regrid_all_channel function that would allow for users to match the sampling rates of a channels in a ds_Sv dataset to a specific channel

[codecov-commenter]

Codecov Report

❌ Patch coverage is 77.69231% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.61%. Comparing base (6cf6cee) to head (87b8944).
⚠️ Report is 43 commits behind head on main.

Files with missing lines	Patch %	Lines
echopype/commongrid/utils.py	70.58%	20 Missing ⚠️
echopype/commongrid/api.py	85.24%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1597      +/-   ##
==========================================
+ Coverage   85.58%   85.61%   +0.03%     
==========================================
  Files          79       78       -1     
  Lines        6998     7169     +171     
==========================================
+ Hits         5989     6138     +149     
- Misses       1009     1031      +22     

Flag	Coverage Δ
integration	80.94% <77.69%> (+0.30%)	⬆️
unit	60.55% <73.84%> (+0.13%)	⬆️
unittests	85.52% <77.69%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[leewujung]

Hey @eliascapriles-NOAA : Thanks for the PR! Below are some comments based on quick look:

• Could you add a few tests for this function? Both unit test using mock data and integration data using a small segment real data would be very useful.
• commingrid/utils.py is meant for util functions used by those in commongrid/api.py, so functions users interact with will be in api.py. What do you think about below?
  • rename regrid_all_channels to regrid in api.py
  • change the argument target_channel_idx to target_channel and require users to put in the channel_id (it's more robust to index with name compared to sequential indices)
  • expand the functionality to include not only specifying a target channel (what you already have) but also allowing specifying a set of target grid? So the function signature would become something like:

  # only one of target_channel or target_grid can be used -- so raise an error if both are provided
  regrid(ds_Sv, target_channel="CHANNEL_ID", target_grid="TARGET_GRID")

[eliascapriles-NOAA]

Sounds good. I will get started on implementing the changes !

[eliascapriles-NOAA]

Hi Wu-Jung sorry for the delay ! The tests uncovered a couple of bugs that I wanted to fix before submitting for a new PR. I have added unit tests for my helper function, and integration test using SV data for the regridding function. Let me know if there are any changes !

[leewujung]

Thanks @eliascapriles-NOAA ! I'm going to ask @LOCEANlloydizard to help review this since you had most of the discussions with him.

@LOCEANlloydizard - Could you please take a look? feel free to ping me for discussions. Thanks!

[eliascapriles-NOAA]

Sounds good ! Thanks @leewujung

[LOCEANlloydizard]

Hey @eliascapriles-NOAA, thx for the PR!
Following our discussion, a small recap (you probably noted more!):

• the errors from CI are now gone with the merge of pandas < 3, just need to address the assertionError
• remove the +20 padding at the end of the function
• double-check that pings are actually aligned before regridding (there’s already a helper in echopype that does this, see getting_started notebook)
• update the notebook to call the function from your PR, so I can run it on my side too
• we can go from there for other points !

One thing I wanted to flag more generally: right now the regridding is done by looping over channel in Python, and inside that running an apply_ufunc that vectorises over ping_time × range_sample.

This is more a design question than a bug: are we happy with looping over channels in Python and vectorising over pings with apply_ufunc, or should we try to make the whole regridding run as one channel-aware vectorised operation? I know you looked into it.., and i could have a look as well! and maybe @leewujung would have recommendations for this?

Cheers!

[LOCEANlloydizard]

This error "TypeError: only integer scalar arrays can be converted to a scalar index" is due to pandas update to version 3.0. Could you merge the latest version of the main echopype repo (pandas is pinned there!)

[leewujung]

Hey @eliascapriles-NOAA @LOCEANlloydizard : Not sure if I am really making a good suggestion, but since apply_ufunc is already configured to be run in parallel, it seems to me that it would be useful to have all channels to be processed at once (instead of in a loop). My thought is then this way the parallelization component is delegated to the xarray-dask combination, rather than us making specific choices ourselves (unless we know for sure that it is better). Let me know what you think!

[eliascapriles-NOAA]

Hi @leewujung LLoyd brought this up yesterday. The reason I currently have the channels in a loop is because my apply_ufunc is parallelizing the function across ping_time as specificed by the Echoview algorithm. However, I will try to rework my function to parallelize across the channel dimension as well

[LOCEANlloydizard]

Hey @eliascapriles-NOAA, thanks for the modifications! I’ve left a few comments above, and also added some broader thoughts below to check first =>

1. Naming: maybe resample would be clearer than regrid, which feels a bit generic? thoughts?

2. Do we actually want the trimming step after the resampling? forgot if we discussed this

• If not, we could remove this entirely
• If yes, we could likely still replace get_valid_max_depth_ping with a xarray based approach?

3. Output dataset:

we now returns a new xarray object which is nice, but we might want to align it more with other regridding like compute_MVBS?

-propagate key variables:
-- water_level (to allow add_depth() downstream)
-- frequency_nominal
-- position variables (lat/lon, etc)

-add provenance:
-- processing_function = "commongrid.regrid" (or resample)?

-add variable-level attrs on Sv:
-- resampling_mode ("target_channel" / "target_grid")
-- target_channel
-- short description?

4. .copy() usage: here (and other places, see comments)

ds_target = ds_Sv.sel(...).copy()
target_range_da = target_grid.copy()

does not seem necessary here since nothing is mutated afterwards? (but could be missing the obvious!)

5. Some small fixes in the tests

• in some places there's a mix of positional indexing and label-based selection? It could be better to consistently use label selection?
• Maybe we can compare the target channel element by element before and after regridding onto its own geometry? should be the same
• If we add water_level, frequency_nominal, lat/lon, attrs, or processing_function, we’ll probably want explicit tests for metadata/provenance too?
• Maybe it would be worth adding a test for cases where target_grid has NaNs/partially valid geometry? just to check the expected behaviour?

It's the final adjusments! Thank you very much for this!
Cheers!

outdated

[LOCEANlloydizard]

Hello @eliascapriles-NOAA, @leewujung, i've made a few final small adjustments! There will probably still be some edge cases to handle over time but I think we're ready to merge 🙂
A few of the final additions:

• support for Sp and TS through the same log -> linear -> log resampling pathway used for Sv
• warning when resampling angle variables
• a few extra tests covering Sp/TS handling, angle warnings, and target-geometry validation

I think it would be nice to eventually add a regression test against Echoview if we can process the same file and match all channels to the 200 kHz geometry, similar to what is shown in the notebook: echostack-org/echopype-examples#101
@eliascapriles-NOAA I can help with this, but I'd need an output from Echoview for this file to compare against.. let me know if you would have time to generate it! Cheers!

modifications applied

[LOCEANlloydizard]

Ok thank you @eliascapriles-NOAA for the file! I’ve added the Echoview regression test using the GitHub asset added in #1679. The comparison between Echoview and echopype shows very close output grids, with Sv differences below 0.003 dB outside the near-transducer region
(See test: echopype/tests/commongrid/test_commongrid_api.py::test_resample_matches_echoview_match_geometry).

We also now raise warnings if users try to resample TS, Sp, or angle variables, since these cases should be interpreted with caution. The notebook has also been updated.
I think we’re good to merge! Cheers