Design software architecture for OGC API Maps

[j08lue]

OGC API Maps is the successor to good old WMS and on the roadmap for implementation in EOEPCA 2.0 data access services.

Its target behaviour is very similar to OGC API Tiles, except that clients request a bounding box instead of tile matrix coordinates. As such, it shares a lot of backend logic with TiTiler, too.

This ticket is to explore how we can provide OGC API Maps on top of TiTiler and/or TiTiler-PgSTAC.

Initial considerations were to use pygeoapi to implement the standard-compliant API and use TiTiler-PgSTAC as a provider.

Note on demoing / validation

OGC API Maps has a few clients already: https://github.com/opengeospatial/ogcapi-maps/blob/master/implementations.adoc, among others for Leaflet, which could be used in a demo of the capabilities.

Acceptance criteria

• Reviewed different viable options
• Documented preliminary architecture design

[jankovicgd]

There are some OGCAPI maps like features in eoAPI already and I am investigating how these can be best exploited and if a mapping can be made through a pygeoapi plugin provider layer. I've outlined this connection in #108 through the following eoAPI requests:

1. /stac/bbox/{minx},{miny},{maxx},{maxy}/{width}x{height}.{format}
2. /cog/bbox/{minx},{miny},{maxx},{maxy}/{width}x{height}.{format}
3. /searches/{search_id}/bbox/{minx},{miny},{maxx},{maxy}/{width}x{height}.{format}

Endpoints 1. and 2. are limited by the fact that they involve image tiles and can span multiple items, which would require stitching. I was thinking of initially dealing with this somehow using MosaicJSON until I @j08lue mentioned /searches in eoAPI. The current approach, using /searches, is as follows:

• a maps query is sent to pygeoapi
• the parameters are checked and are forwarded to the provider plugin
• the provider has the following logic tied to eoAPI
  • the search is registered based on query params (/searches/register)
  • the search is immediately retrieved (endpoint 3.)
• the retrieved image is forwarded as a response

This approach has as a benefit further of supporting and connecting open tools such as pygeoapi & eoAPI, and avoiding to create same features twice. The plugin can be maintained for the duration of the project and, if it proves promising and useful, could later be maintained by the community.

Noting also some drawbacks to this approach:

• Frequent use of the /searches endpoint with a wide array of bounding boxes may create numerous entries in the eoAPI database, many of which will realistically be queried only once.
• The approach involves larger network overhead. Ideally, there should be a single request-response interaction between the client and server. However, in this case, the workflow involves multiple interactions: request->request-response->request-response->response. This additional complexity may lead to higher latency and reduced efficiency.
• Any increased load on eoAPI will also affect the provider.
• Error propagation. Area for errors is increased so we have to note how eoAPI errors map to OGCAPI maps errors, so that the client/user can best decide how to deal them.

Any input is greatly appreciated. @vincentsarago @Schpidi @pantierra @kalxas @jonas-eberle

[j08lue]

Great to see your work towards OGC API Maps via pygeoapi in https://github.com/EOEPCA/eoapi-maps-plugin, @jankovicgd.

@vincentsarago made me aware that adding Maps support directly in TiTiler has been discussed in the past, too:

• OGC API - Maps  developmentseed/titiler#753

Have you considered this, actually?

[j08lue]

We reviewed the current architecture approach that @jankovicgd has been pursuing:

POC of a pygeoapi based architecture

1. pygeoapi provides the Maps interface
2. pygeoapi has instance-level configuration for which collections it exposes
3. pygeoapi calls eoAPI's TiTiler /search and /bbox to render the maps

Demo

Nicely prepared by @jankovicgd:

1. Application for Maps with pygeoapi: https://github.com/EOEPCA/eoapi-maps-plugin
2. EOEPCA service: https://github.com/EOEPCA/eoepca-plus/blob/deploy-develop/argocd/eoepca/data-access/parts/app-data-access-maps.yaml
3. Configuration (including resources definition): https://github.com/EOEPCA/eoepca-plus/blob/ee0ed6ec08ca84366a21aa3b86a3249893a4f00a/argocd/eoepca/data-access/parts/values/values-eoapi-maps-plugin.yaml#L95-L121
4. Service landing page: https://maps.develop.eoepca.org/
5. Sample request: https://maps.develop.eoepca.org/collections/sentinel-2-l2a/map?bbox=18.5,52.3,18.6,52.4&datetime=2023-12-31T00:00:00Z/2023-12-31T23:59:59Z (NB: slow and flaky because of the JPEG2000 Sentinel-2 assets)

Issue: no dynamic specification of resources

While the POC works nicely, an issue is that resources have to be configured at application level - i.e. users cannot request Maps for collections that are not yet configured in the application and that configuration requires operator access to Kubernetes. As @kalxas pointed out, this approach is similar to how GeoServer classically works.

In contrast, the classic MapServer approach, which is also TiTiler's, is that resources can be specified dynamically - either by user input of an asset URL accessible to the service, or via STAC collection/item (TiTiler-PgSTAC). Also wit this approach, restrictions can be imposed on which resources a user can access via private STAC collections or application-level enforcement logic.

Decision

Arguments from one or the other approach can be derived from target use cases - do EOEPCA platform owners require dynamic configuration of resources or not?

Another consideration is consistency with the other (eoAPI) data access services. Maps should not behave differently from Tiles.

Conclusion: Let us see whether it is feasible to implement Maps support directly in TiTiler, as discussed above.

[jonas-eberle]

Arguments from one or the other approach can be derived from target use cases - do EOEPCA platform owners require dynamic configuration of resources or not?

Yes, terrabyte would need dynamic configuration of resources. E.g., when a new collection is being added to the STAC API, this should be automatically available in the OGC Maps service.

In addition, it would be nice to have that a user can reference an individual STAC item or geospatial file (e.g., COG) without the need to register it to a STAC API (similar to TiTiler's functionality).

[jankovicgd]

There is https://docs.pygeoapi.io/en/latest/admin-api.html which may be of use for dynamic generation. I'm investigating

[kalxas]

WMO is using pygeoapi in production and dynamic generation of collections is supported by storing the pygeoapi configuration in a database. More details to follow.

[jankovicgd]

Investigate automatic syncing of STAC to pygeoapi via admin api. Reasoning: the user experience may be a bit tedious if they would need to create a very similar request twice to different endpoints. Periodically sync and maybe take a look at adding an ogc STAC extension.

@jonas-eberle one thing that is a bit unclear since there are also the registration, processing and datacube BBs. How is the user creating collections/adding data? Is this also handled by the processing BB/registration BB?

The second point is solved via /cog & /stac but i notice that these are missing in the deployment, I was using them for some tests: https://titiler.xyz/api.html#/ @j08lue

[jankovicgd]

Also look at: https://github.com/stac-extensions/render

[jankovicgd]

After reading through the extension, tinkering with it and a little help from @fabricebrito the best approach is to avoid adding a new extension and use/work on this. So in the end the user creating the collection, and wanting to render it, would need to use the extension.

For default rendering (eg. tci) the user should create a render with the default key and the rest of the renders will be exposed as styles, and maybe try to use the visual asset as a fallback before failing.

@kalxas there is some work going on with ogcapi-styles in pygeoapi according to geopython/pygeoapi#1809, do you think this clashes in some sort of a way? From my tinkering with pygeoapi it seems straight-forward to just use the render extension

[j08lue]

As discussed, let us consider not requiring a render that has a specific key ("default") and instead use the first one in the (ordered) mapping.

[jankovicgd]

I'll add this as an additional check before the fallback.

1. Check if default is present
2. Select the first render if not there
3. Fallback to visual asset
4. Fail