Sentinel-1 GRD coordinates and data type inconsistency with EOPF sample notebooks

[beatrizbsperes]

The zarr samples used on the EOPF sample notebooks don't share the same data format as the ones available on EOPF Sentinel Zarr Samples Service STAC API. For Sentinel-1 GRD catalog we face the following issues:

• On the EOPF sample notebooks we can see that the units for azimuth_time is datetime64[ns]while or the STAC zarr samples available for users the azimuth_time units is uint16
• For grd, under measurements group, following EOPF sample notebooks we can see that we have 4 coordinate types (azimuth_time. ground_range. line and pixel) while for the STAC zarr samples available for users we only see 1 coordinate type, azimuth_time
• the datasets used on the EOPF sample notebooks for Sentinel-1 GRD is not available on the EOPF Sentinel Zarr Samples Service STAC API, which makes it impossible for users to follow the same workflow since the availabe data doesn+t follow the same structure

this is the grd data array shown on EOPF sample notebooks (using the unavailable product):

<xarray.DataArray 'grd' (azimuth_time: 16694, ground_range: 26239)> Size: 876MB
dask.array<open_dataset-grd, shape=(16694, 26239), dtype=uint16, chunksize=(2557, 26239), chunktype=numpy.ndarray>
Coordinates:
  * azimuth_time  (azimuth_time) datetime64[ns] 134kB 2017-05-08T16:48:30.467...
  * ground_range  (ground_range) float64 210kB 0.0 10.0 ... 2.624e+05 2.624e+05
    line          (azimuth_time) int64 134kB dask.array<chunksize=(16694,), meta=np.ndarray>
    pixel         (ground_range) int64 210kB dask.array<chunksize=(26239,), meta=np.ndarray>
Attributes:
    _eopf_attrs:  {'coordinates': ['azimuth_time', 'line', 'pixel', 'ground_r...
    dtype:        <u2
    long_name:    measurement data set for GRD IW

this is the grd data array got when using the zarr samples available for users on the STAC API:

<xarray.DataArray 'grd' (azimuth_time: 16695, ground_range: 26006)> Size: 868MB
dask.array<open_dataset-grd, shape=(16695, 26006), dtype=uint16, chunksize=(5161, 26006), chunktype=numpy.ndarray>
Coordinates:
  * azimuth_time  (azimuth_time) uint16 33kB 19608 9752 65432 ... 61727 51872
Dimensions without coordinates: ground_range
Attributes:
    _eopf_attrs:  {'coordinates': ['azimuth_time'], 'dimensions': ['azimuth_t...
    dtype:        <u2
    long_name:    measurement data set for GRD IW

[clausmichele]

Hi @beatrizbsperes, you are right and we are aware of that. The data used in the notebook has been converted using an unreleased version of the eopf-cpm tool to convert the data, which fixed the issues we reported here: https://gitlab.eopf.copernicus.eu/cpm/eopf-cpm/-/issues/716

Unfortunately, those fixes didn't get to a release yet (you can read the comments at the end of that issue). As soon as an updated release of eopf-cpm will be available, the data will be converted again and the STAC Collection updated (see EOPF-Sample-Service/eopf-sample-data#8 (comment))

[emmanuelmathot]

Thank you @clausmichele for the information. Have you got more samples translated with the fix for GRD? We actually need some over a area before and after a flooding event. Should @beatrizbsperes register an issue in https://github.com/EOPF-Sample-Service/eopf-sample-data?

[clausmichele]

@emmanuelmathot no we don't have other updated products. Maybe @christophreimer could help here!
Anyway, if you want to use GRD for flood detection, do you already know how to compute the backscatter starting from the Zarr product? I didn't have time to focus on that but it would be particularly interesting for many use cases.

[beatrizbsperes]

@clausmichele we are still trying to figure it out but we found this library called Sarsen that might be pretty useful (https://github.com/bopen/sarsen). What are you ideas so far?

[clausmichele]

@clausmichele we are still trying to figure it out but we found this library called Sarsen that might be pretty useful (https://github.com/bopen/sarsen). What are you ideas so far?

Could be a good idea! Bopen is also the company behind the EOPF Zarr Sentinel-1 converters, it would be worth to ask them if they know how much work would be to extend the compatibility to Zarr.

My idea was to implement sigma0 backscatter computation in Python directly without additional libraries, which would be a simpler processing compared to the products that can be computed using libraries like sarsen (radiometrically corrected gamma0).

[beatrizbsperes]

@clausmichele Actually sarsen it's already compatible with zarr, from my understanding but I haven't had time to dive deeper on the library.

That's interesting, it might work as well!

[clausmichele]

Hi @beatrizbsperes I wanted to give you an update on this issue:

• I tried SARSEN but didn't manage to do much, see gtc computation succeeds but result is not meaningful bopen/sarsen#79
• We will update the conversion workflow to eopf 2.6.1 soon, so that the Sentinel-1 data published on STAC is finally usable. I'll let you know once this happens.

[beatrizbsperes]

Hello @clausmichele ! Thanks for the update! It's good that you're being able to use sarsen (I tried but I couldn't perform any operation using .zarr) but still tricky to see the final result after the gtc... Did you use .safe format, am I right?

[clausmichele]

Hello @clausmichele ! Thanks for the update! It's good that you're being able to use sarsen (I tried but I couldn't perform any operation using .zarr) but still tricky to see the final result after the gtc... Did you use .safe format, am I right?

Well, I wouldn't say I was able to use it actually, since I tried the basic example they provide with SAFE but didn't get any meaningful result :)