- Status: approved
- Last Updated: 2022-06-01
- Objective: determine which Google Cloud (GCP) projects delivery testing systems should run against
Thoroughly testing our Delivery component requires having test projects that we can (attempt to) deploy Emblem to.
These projects must be created somehow.
- Some GCP environments have IAM policies prohibiting users from creating new projects.
- Testing against ephemeral projects might lead to:
- fewer flakes
- less dependence on pre-existing project state
- Creating ephemeral projects programmatically requires allowing non-human IAM principals (such as Service Accounts) to create new GCP projects.
- Option 1: Pre-provision long-lived projects manually beforehand
- Option 2: Create ephemeral projects programmatically at test time
We chose to pre-provision long-lived projects manually beforehand.
This was largely due to organization-level constraints that make it difficult (if not impossible) for our team to programmatically create new GCP projects.
This may cause a higher degree of CI pipeline flakiness, in both directions.
- False passes may occur if tests depend on GCP resources within a project that are undeclared in Terraform.
- False failures may occur if the project's previous state breaks our tools'/Terraform's attempts at automated state management.
In our experience as a team, we've found that managing pools of long-lived GCP projects can be complicated.
Without pooling or ephemeral project creation, there's no way to parallelize multiple presubmit builds.
Thus, we decided to avoid presubmit builds for Delivery testing and use nightly builds instead.
At the time of writing, the main reason to revise this decision would be if we decide that Delivery tests should be ran as presubmits.
Related Issue