Sentinel 2 Demystifying Zarr

[vdumoul]

In the Context of Sentinel 2 we have an important user that does its own atmospheric correction from L1C to L2a. He has some doubts on the potential updates of the format:

• Overlap removal: In the long term we want to avoid having the overlap stored in the product. It would be good to demonstrate how to get a subset of the neighbours of L1C products
• Scalability issues: how to provide in parallel several large datacube.

The propose works to do are the following:

1. Overlap removal

• Show that a 10 km neighbourhood can still be queried easily and quickly for any tiles.
  • This is a classic Zarr request
  • Provide pixel wise equality checks between SAFE and Zarr outputs for at least one representative tile and date.
  • We shall demonstrate that the query of the surrounding neighbours at the end means not so much additional work. This could show how to enlarge the initial tile.
  • If we have the UTM product, we could even simplify the full extraction: but this is not guaranteed to be our future product
  • Final output would be a function to recreate the overlap like legacy product
• Try to demonstrate that one of his workflows give identical results to the SAFE format (seems obvious as we will have the same inputs, but it would make more sense than simply extracting the overlap):
  • Cloud detection + shadow pairing
  • Atmospheric adjacency (blur) correction
  • Edge aware ML segmentation/classification (this is maybe something we already have)

2. End to end continental throughput

• Measure time and resources needed to extract five 1-full-month temporal cubes covering ≈ 2 500 European tiles (100 × 100 km each), all launched concurrently.
• Compare wall clock and I/O cost with today’s straight SAFE file download.
• The example could add some index computation trying to demonstrate the usefulness of the cloud approach with respect to the classic download and calculate
• Quantify server side compute load for the above extraction pattern; confirm the platform can sustain multiple users issuing similar requests without bottlenecks.

[przell]

#100 and #104 will work on some of these requirements. Good to have this list sothat we can include the requirements while developing.

[przell]

@vdumoul, here's some progress on the benchmarking comparing partial reads of eopf vs safe. Still WIP and needs some discussions. But still might be interesting to follow: #104 (comment)