Add notification of nightly test failures

[grayside]

Proposal

Add notification of nightly test failures. Email is okay, but preference would be to open issues, and not open a new issue if an existing test failure issue is open.

Problems this will solve

Ensuring test failures are in the team's work queue and dashboards without daily manual review.

[ace-n]

TODO: @ace-n to collaborate with @glasnt and see what the best approach is.

(Context: currently, our CI processes run in Cloud Build.)

[ace-n]

This is 🚨 blocked 🚨 on this PR

That PR has since been merged, so this is unblocked.

Ideally, I'd implement this collaboratively with someone else. (It's a great opportunity excuse to learn how Emblem's CI systems work!)

UPDATE: @rogerthatdev has expressed interest in working on this; I'll set up a sync w/ them internally.

[ace-n]

There are currently three nightly tests running in our Emblem setup, per our testing documentation:

1. Content API
2. Installation Test
3. Website

@rogerthatdev and I will create separate PRs to enable build failure notifications for each.

Relevant links: docs / base PR

[ace-n]

@rogerthatdev and I met today

Takeaway
We should document the "plugin model" used by the Installation tests.

These notifiers can live in the /cloud-build-notifiers category

---

Next steps

• [Issue Comment (optional)] Identify the TF resources that we need based on this tutorial
• [PR 1, Setup] Put those resources in a PR in their own "plugin"
• [PR 2a, Setup] Add required Dockerfile contents
• [PR 2b, Setup] Add CD build for Cloud Build notifier image
• [PR 3, Execution] Get everything working end-to-end (i.e. perform any bugfixes)

---

Notes from our August 1st meeting:

• We'll need to create a new google_storage_bucket
• Put files in a /cloud-build-notifiers "plugin" directory within the repo root
• For now, only support notifications for nightly tests
• Dedupe resources where we can (i.e. have one build notification setup for multiple builds, if practical)

[ace-n]

Assigning @rogerthatdev since we've synced on this.

(Roger - as always, feel free to ask me any questions you might have! 🙂 )

[ace-n]

Downgrading to P2, since this is not required for our next major milestone.^ (It is a fast follow, however.)

^ AKA Public Preview