Nightly monitoring & reporting: staging matches HEAD

[grayside]

Proposal

In order to ensure our staging CD system is working correctly on an ongoing basis, we should verify that the staging environment has deployed the latest commit to the main branch (HEAD).

This will require a multi-step process:

1. Add Git versioning data to the app in one or more places
2. Add monitoring to compare the app to the GitHub repository
3. Add reporting to file / reopen a GitHub issue if there is a mismatch

Initial Implementation Thoughts

Add Git Versioning Data to the App

Plumbing git revision metadata through how the container image is built.

Because our deploy process sits behind an Artifact Registry Pub/Sub message, we may need to add this as build metadata to the container image itself. This is often done with container labels, such as https://docs.docker.com/config/labels-custom-metadata/.
If we do so, we may want to investigate using a standardized key scheme such as https://github.com/opencontainers/image-spec/blob/main/annotations.md.

When deploying the application, we'll want to access this label from inside the container. This may not be possible, but in Cloud Build we can inspect the container, pull out the label, and convert that to an environment variable on the Cloud Run service.

When deployed with the deploy commit data, include it in a healthcheck endpoint.

Monitoring

Verify the API and Website healthcheck endpoints report a Git version that matches the repository.

Notifications

We're currently exploring GitHub issue notifications in #414 , perhaps the mechanism used there could be used here?

Why Not Monitor Failed Deploys?

We previously had an incident where our deployment triggers stopped firing. In that case, we wouldn't have a failed deployment to verify.