STAC ingestion workflows

[hrodmn]

Let's have a discussion about STAC metadata generation and ingestion workflows. There are lots of ways to do this work but most of the existing implementations are highly specific to a particular cloud environment and the components are not easily recyclable. I think it would be useful to have a discussion about the design of a STAC ingestion ecosystem so that we can start to assemble the building blocks in a way that could serve multiple projects/users.

[hrodmn]

I need to build an event-based STAC ingestion workflow for MAAP in the next few weeks but before I retreat to a cave and add something to maap-eoapi I thought I would start a conversation here!

This diagram represents how I am thinking about STAC metadata generation/ingestion pipelines in a generic, architecture-agnostic way (i.e. serverless vs kubernetes, pgstac vs geoparquet backend, etc.). This is heavily based on the awesome capabilities of stactools-pipelines. My goal is to create a stack with some kind of API at the top (Ingestor API) that provides users and client applications with entrypoints for adding work into the item-gen or item-load queues, but I would want to make it possible to trigger these workflows with events (like uploading COGs or catalog.json to a bucket in S3).

Definitions

• item-gen: generate STAC item metadata for asset(s)
  • using stactools packages or rio-stac or some custom process
• item-load: load STAC metadata into a STAC catalog
  • layer to limit the flow of incoming data into a STAC backend
• STAC backend: pgstac, stac-geoparquet, etc
• events:
  • e.g. item-gen: you want to initiate an item-gen process for every file that gets uploaded to a storage container
  • e.g. item-load: you want to listen for STAC items that land in a bucket so you can load them into a pgstac database

Discussion points

• Why not route everything through the Ingestor API for synchronous validation?
  • Maybe we should! Checking to see if the collection exists for an item and for compliance with the STAC spec should be prerequisites to running something like pypgstac load on an item.
  • But what if an async workflow would be fine? We need systems that can run totally asynchronously by listening for files landing in a bucket. In a setup where the only entrypoint is the Ingestor API would we trigger the Ingestor API function, then trigger the item-load/item-gen function in a second hop?
  • I worry about forcing everything through a single homemade API because we might hit concurrency limits or run into other scale limitations in some scenarios (e.g. default concurrent execution limit for Lambda is 1000). I like the idea of using a notification system as an API because we can trust APIs like aws s3 cp to work really well for many clients.
  • Committing to a mostly async workflow will require some mechanism for exposing failed tasks
• How can we logically separate the codebases for these components?

cc @alukach @sharkinsspatial