Skip to content

fix(ci): serialize AWS and GCP Terraform state writers on environment, not branch - #1812

Merged
cristim merged 1 commit into
mainfrom
fix/1806-serialize-aws-gcp-tfstate
Aug 13, 2026
Merged

fix(ci): serialize AWS and GCP Terraform state writers on environment, not branch#1812
cristim merged 1 commit into
mainfrom
fix/1806-serialize-aws-gcp-tfstate

Conversation

@cristim

@cristim cristim commented Aug 12, 2026

Copy link
Copy Markdown
Member

Closes #1806

Both halves of #1801 were fixed for Azure only in #1803. This lands the equivalent for AWS and GCP. It is not a copy-paste of #1803: the environment is derived differently per workflow, Fargate uses a separate state namespace and needs its own group, and one lock-removal step is deliberately kept.

Enumeration, not sampling

.github/workflows/ holds 16 .yml files (a 17th entry is README.md). All 16 were classified by grepping for backend config, state keys, bucket names and state-mutating terraform subcommands, not by filename.

file verdict evidence
aws_sanity.yml not a writer rg -l -i 'terraform|tfstate|aws s3|gsutil' -> no match
azure_sanity.yml not a writer same, no match
frontend-build.yml not a writer same, no match
frontend-build-sentinel.yml not a writer same, no match
frontend-e2e.yml not a writer same, no match
ci.yml not a writer 30 terraform mentions, all terraform init -backend=false + validate at :386. No backend, no lock
pre-commit.yml not a writer 17 mentions, all prose in comments about the terraform_validate hook
database-migration.yml not a writer terraform init at :258, :355, :432 then terraform output only. output takes no lock. (Those are bare init with no -backend-config against a partial backend block, flagged in #1803 and still unfixed; either way nothing is written)
deploy-all.yml not a direct writer one terraform mention, in a comment. Reaches state only via uses:
deploy-azure.yml Azure only already fixed in #1803, out of scope
deploy-aws-lambda.yml writer terraform apply :281; key github-<env>/terraform.tfstate
deploy-aws-fargate.yml writer terraform apply :179; key github-fargate-<env>/terraform.tfstate
deploy-gcp.yml writer terraform apply :170; prefix github-<env> (GCS)
destroy-fargate-dev.yml writer terraform destroy :143; key github-fargate-dev/terraform.tfstate
cleanup-staging.yml writer x4 terraform destroy :169, :246, :316, :412 plus four terraform state rm at :395-400
rollback.yml writer x4 terraform apply :238, :326, :389, :481

Line numbers above are origin/main at 17a568f42.

This differs from the issue. The issue names three workflows for half 1 and two steps for half 2. The real candidate set is larger in two places and smaller in one:

  • destroy-fargate-dev.yml is a state writer the issue does not mention, and it carried both defects: no concurrency group at all, and an unconditional aws s3 rm of the .tflock before every terraform init (:111-114).
  • deploy-gcp.yml had two lock deletions, not one. The issue cites the failure() || cancelled() step. There was also an unconditional gsutil rm inside Terraform Init (:145-150) that deleted the lock on every single run, before init, with no condition whatsoever. That is strictly worse than the one the issue names.
  • deploy-aws-fargate.yml no longer has half 2. The issue lists it under half 1 only, which is right, but for completeness: its lock-removal step (:138-152) is already gated on inputs.clear_stale_lock == true, a workflow_dispatch-only boolean defaulting false, and its failure-path release step was already deleted with a comment explaining why. That step is retained (justified below).

Enumerated mechanically rather than by eye: a script parses every workflow, selects jobs whose run: blocks contain a state-mutating terraform subcommand, and prints the group next to the state key.

origin/main:  jobs mutating terraform state: 13   without a group: 10
this PR:      jobs mutating terraform state: 13   without a group: 0
              with cancel-in-progress true: 0

Half 1: group on the job that writes state, keyed on the value that builds the state key

Workflow-level concurrency cannot see needs (constraint proven below), so each group lives on the deploying job. Three distinct state objects across AWS and GCP, therefore three group prefixes:

group state object
aws-tfstate-<env> s3://<bucket>/github-<env>/terraform.tfstate
aws-fargate-tfstate-<env> s3://<bucket>/github-fargate-<env>/terraform.tfstate
gcp-tfstate-<env> gs://<bucket>/github-<env>/default.tfstate

Fargate gets its own prefix rather than sharing aws-tfstate-*. Sharing would serialize two independent state files against each other, buying nothing and costing queue evictions.

Every group and the state key it guards, side by side, at post-merge line numbers:

file job group state key expression same value?
deploy-aws-lambda.yml build-and-deploy :222 aws-tfstate-${{ needs.prepare.outputs.target_environment }} :255 ENVIRONMENT: needs.prepare.outputs.target_environment -> :257 key = "github-%s/terraform.tfstate" yes
deploy-aws-fargate.yml deploy :132 aws-fargate-tfstate-${{ needs.prepare.outputs.environment }} :175 ENVIRONMENT: needs.prepare.outputs.environment -> :177 key = "github-fargate-%s/terraform.tfstate" yes
deploy-gcp.yml build-and-deploy :131 gcp-tfstate-${{ needs.prepare.outputs.environment }} :165 ENVIRONMENT: needs.prepare.outputs.environment -> :167 prefix = "github-%s" yes
destroy-fargate-dev.yml destroy :85 literal aws-fargate-tfstate-dev :127 literal github-fargate-dev/terraform.tfstate yes
cleanup-staging.yml destroy-aws-lambda :106 literal aws-tfstate-staging :137 literal github-staging/terraform.tfstate yes
cleanup-staging.yml destroy-aws-fargate :191 literal aws-fargate-tfstate-staging :222 literal github-fargate-staging/terraform.tfstate yes
cleanup-staging.yml destroy-gcp :345 literal gcp-tfstate-staging :379 literal prefix = "github-staging" yes
rollback.yml rollback-aws-lambda :186 aws-tfstate-${{ inputs.environment }} :240 ENVIRONMENT: inputs.environment -> :244 key = "github-%s/terraform.tfstate" yes
rollback.yml rollback-aws-fargate :288 aws-tfstate-${{ inputs.environment }} :342 ENVIRONMENT: inputs.environment -> :346 key = "github-%s/terraform.tfstate" yes, see below
rollback.yml rollback-gcp :368 gcp-tfstate-${{ inputs.environment }} :412 ENVIRONMENT: inputs.environment -> :417 prefix = "github-%s" yes

That is 10 AWS/GCP state-mutating jobs, every one of them previously ungrouped.

rollback-aws-fargate takes the Lambda group on purpose

Despite its name, that job's backend key is github-<env>/terraform.tfstate, the Lambda namespace, not github-fargate-<env>/. Giving it aws-fargate-tfstate-* would have serialized it against a state file it never touches while leaving the one it does write unguarded, which is exactly this bug reproduced in a new place. So the group names the object the job actually locks.

The underlying namespace mismatch is a real pre-existing defect (a Fargate rollback applies compute_platform=fargate into the Lambda state). It is filed as #1811 rather than changed here, because moving the key changes which infrastructure a rollback rewrites and may strand resources already written to the wrong file. A comment at the job says the group moves when #1811 lands.

Workflow-level concurrency removed from all three deploy workflows

deploy-lambda-${{ github.ref }}, deploy-fargate-${{ github.ref }} and deploy-gcp-${{ github.ref }} are deleted, replaced by a comment saying why the level had to change. Keeping them would imply a state guard they never provided, and they group runs the job-level key deliberately does not (a prod dispatch and a main push write different state files and have no reason to queue behind each other). Same call as #1803 made for Azure. The image build is unaffected: tags derive from the git commit, so distinct refs produce distinct tags and same-ref runs were never racing on a tag.

deploy-all.yml gets no group, only a comment

It reaches AWS and GCP state exclusively through uses:, and the called workflow's deploying job runs as a real job of that run and is serialized there. A group on the caller job would deadlock: the caller holds the group while waiting on the inner job queued behind it. #1803 documented this for deploy-azure; the note is generalized to all four callers.

Half 2: four lock deletions removed, one retained

file step disposition
deploy-gcp.yml unconditional gsutil rm of .tflock inside Terraform Init deleted
deploy-gcp.yml Release state lock on failure, if: failure() || cancelled() deleted
deploy-aws-lambda.yml Release state lock on failure, if: failure() || cancelled() deleted
destroy-fargate-dev.yml unconditional aws s3 rm of .tflock inside Terraform Init deleted
deploy-aws-fargate.yml Clear stale state lock (operator-triggered only), if: inputs.clear_stale_lock == true retained

Deleted rather than made conditional, per #1803's decisive argument: cancelled() steps run while terraform apply is still gracefully shutting down and may still be writing state, so the step destroys a lock the dying run is still using. The failure() half is no better: a run that failed because it could not acquire the lock deletes the lock held by the run that is still applying. Neither step checked age, and neither checked the lock was this run's. A loud Error acquiring the state lock is the correct outcome of a real collision.

The two unconditional pre-init deletions were worse than either: they fired on every run regardless of outcome, so any second writer wiped the first's live lock before even trying to acquire one.

The retained step is justified: clear_stale_lock is a workflow_dispatch-only boolean defaulting to false, undeclared on the workflow_call path (so it renders null and the == true guard is false). It only fires when an operator explicitly asks for it, having confirmed the owning run is dead. That is the operator-confirmed recovery shape this issue asks for, not the blanket delete it asks to remove. It also gives the four deleted steps somewhere to point: every replacement comment names terraform force-unlock <ID> and runbooks/terraform-stuck-lock.md.

Trade accepted, stated plainly: a Terraform crash that strands a lock now wedges the pipeline until someone runs force-unlock, instead of the next run silently clearing it. That is the point.

Verification

check exit result
actionlint on all 7 touched files, this branch 1 352 lines, 88 findings
actionlint on the same 7 files at origin/main 1 352 lines, 88 findings
diff of the two, normalized to strip file:line:col prefixes and source gutters 0 byte-identical. Zero new findings. Every finding is a pre-existing shellcheck SC2086/SC2129 info/style note on steps this PR does not touch
negative control: the job-level expression moved to workflow-level concurrency in a scratch copy of deploy-gcp.yml 1 context "needs" is not allowed here. available contexts are "github", "inputs", "vars"
pre-commit run --files <7 files> 0 all applicable hooks pass

actionlint is not wired into CI or pre-commit, so it gates nothing either way.

The negative control is what makes the clean run evidence rather than silence: it proves actionlint actually enforces context availability for concurrency, so a clean result on the job-level form means the expression resolves, not that nothing was checked.

The group suffix can never be empty, established by execution

An empty suffix would collapse every run into one group, a new bug. A script extracts each prepare job's set-env script verbatim from the committed YAML and executes it under 12 input combinations, 36 executions across the three deploy workflows:

deploy-aws-lambda.yml -> group aws-tfstate-<suffix>
  event=push              input=''        -> group=aws-tfstate-dev
  event=workflow_dispatch input='dev'     -> group=aws-tfstate-dev
  event=workflow_dispatch input='staging' -> group=aws-tfstate-staging
  event=workflow_dispatch input='prod'    -> group=aws-tfstate-prod
  event=release           input=''        -> group=aws-tfstate-prod
  event=release           input='staging' -> group=aws-tfstate-staging
  event=pull_request      input=''        -> prepare EXITS 1 -> job skipped, group never claimed
  event=schedule          input=''        -> prepare EXITS 1 -> job skipped, group never claimed
  event=workflow_dispatch input=''        -> prepare EXITS 1 -> job skipped, group never claimed
  event=push              input='bogus'   -> prepare EXITS 1 -> job skipped, group never claimed
  event=push              input=' '       -> prepare EXITS 1 -> job skipped, group never claimed
  event=push              input='DEV'     -> prepare EXITS 1 -> job skipped, group never claimed

empty-suffix occurrences: 0   (across all three workflows, 36 executions)

deploy-aws-fargate.yml is stricter still: it has no push or release trigger, so every empty-input row exits 1.

What guarantees this is the dev|staging|prod allowlist under set -euo pipefail, not the default arms. Empty string does not match, falls to *), and exits 1. Each deploying job then declares needs: prepare with no if: (verified programmatically across all 13 state-mutating jobs, printed alongside the group), and a job whose needs failed is skipped, so it never dispatches and never occupies a group. Job-level concurrency is evaluated at dispatch, after needs resolves, so there is no ordering in which an empty value claims a group.

The three rollback.yml jobs and the four literal-suffix jobs cannot be empty by construction: inputs.environment is a required choice of dev|staging|prod on rollback.yml's only trigger (workflow_dispatch, no workflow_call), and the rest are literals.

One vocabulary across writers

Groups only serialize against each other if the same environment produces the identical string. All ten AWS/GCP suffixes resolve to exactly dev, staging or prod: from prepare's runtime allowlist (deploy workflows, needed because workflow_call inputs are free-form strings), from a type: choice (rollback.yml), or from a literal (cleanup-staging.yml, destroy-fargate-dev.yml). No development/dev divergence is reachable.

What remains unproven

  • The concurrency behaviour itself is not empirically demonstrated, and this is the single largest gap. Two overlapping live runs were not staged; group semantics are only observable on GitHub's runners. The evidence is that the expressions are valid, in scope, evaluate to the intended string on every path, and name the object each job actually locks. That is not evidence GitHub queued anything. Since the failure mode is silent, a fix that merely makes collisions rarer would not be a fix, so this gap matters.
  • Nothing exercises most of these groups until someone dispatches the workflow. A push to main exercises aws-tfstate-dev and gcp-tfstate-dev. deploy-aws-fargate.yml, destroy-fargate-dev.yml, cleanup-staging.yml and rollback.yml are all dispatch-only and will not be touched by CI on this PR. The first real cross-workflow collision after merge is the actual test.
  • A queued rollback or destroy can be evicted. GitHub keeps one pending entry per group; cancel-in-progress: false protects the running job, not the pending one. A rollback-aws-lambda queued behind an in-flight prod deploy is cancelled if another deploy queues into the same group. rollback.yml's summary job exits 1 on any non-success, so that surfaces; cleanup-staging.yml and destroy-fargate-dev.yml have no aggregator, so an evicted destroy shows only as a cancelled job. Inherent to sharing one group across workflows, and still better than concurrent writers.
  • Interaction with environment: bindings is untested. Nine of the ten grouped jobs are bound to deployment environments (all but deploy-gcp.yml's build-and-deploy). If required-reviewer rules are ever configured (per sec(ci): deployment environments have no protection rules, so environment-bound credentialed jobs run unapproved #1660, none exist today), it is unclear whether a job parked awaiting approval holds its group. If it does, an unapproved rollback blocks deploys to that environment for the approval window. Same unknown fix(ci/azure): serialize Terraform state writers on environment, not branch #1803 recorded; not resolvable from the docs and not tested.
  • Nothing enforces the group prefixes staying in sync. The aws-tfstate-*, aws-fargate-tfstate-* and gcp-tfstate-* literals are matched by convention. A future writer added without the group would silently not serialize, the same class of silent failure as the original bug. Nothing enforces the shared azure-tfstate-* concurrency group across the three state writers #1807 already tracks this for Azure; it now covers three more prefixes.
  • terraform plan also takes a state lock, so plan-only paths could in principle contend. None exist here: every workflow that plans also applies, in the same job, inside the same group. Noted rather than assumed away.
  • I did not verify the backend buckets or the live state objects. The mapping from backend config to object path is read from the printf templates, not from the buckets. TF_BACKEND_* are secrets and were not resolved.

Follow-up filed

Summary by CodeRabbit

  • Improvements

    • Deployment, cleanup, and rollback operations now coordinate more reliably when targeting the same environment.
    • Independent environments can continue deploying in parallel without unnecessary cancellation.
    • In-progress operations are preserved instead of being automatically canceled.
  • Operational Changes

    • Automatic removal of infrastructure locks has been disabled to prevent potential state corruption.
    • If a stale lock occurs, recovery now requires explicit operator intervention.

…, not branch

Both halves of #1801 were fixed for Azure only in #1803 and were still live on
AWS and GCP.

Half 1: the state key is built from the environment but the concurrency group
was keyed on `github.ref`, so two runs on different refs that resolve to the
same environment landed in different groups and applied against one state file.
Workflow-level `concurrency` cannot see `needs`, so each group moves to the job
that writes state, keyed on the same value that builds the state key:

  aws-tfstate-<env>          github-<env>/terraform.tfstate          (S3)
  aws-fargate-tfstate-<env>  github-fargate-<env>/terraform.tfstate  (S3)
  gcp-tfstate-<env>          github-<env>/default.tfstate            (GCS)

Applied to all ten previously ungrouped state-mutating jobs across
deploy-aws-lambda.yml, deploy-aws-fargate.yml, deploy-gcp.yml,
destroy-fargate-dev.yml, cleanup-staging.yml and rollback.yml, so serialization
holds across workflows, not just within one. `cancel-in-progress: false` on
every one: cancelling mid-apply leaves a half-applied stack and a stuck lock.

Half 2: four steps deleted the state lock object with no age check and no check
that the lock was this run's. Two ran unconditionally before `terraform init`,
two on `failure() || cancelled()`. The `cancelled()` half is the decisive one:
those steps run while `terraform apply` is still shutting down, destroying a
lock the dying run may still be using. All four are removed rather than made
conditional, so a real collision fails loudly with "Error acquiring the state
lock". deploy-aws-fargate.yml's operator-gated `clear_stale_lock` step is kept
as the recovery path.

destroy-fargate-dev.yml was not named in the issue but writes
github-fargate-dev/terraform.tfstate and carried both defects.

rollback.yml's rollback-aws-fargate takes the aws-tfstate-* group because its
backend key is the Lambda namespace, not the Fargate one. That pre-existing
mismatch is tracked in #1811.

Closes #1806
@cristim cristim added triaged Item has been triaged priority/p1 Next up; this sprint severity/high Significant harm urgency/this-sprint Within the current sprint impact/internal Team-internal only effort/m Days type/bug Defect labels Aug 12, 2026
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: eb1f0562-9464-406d-bacd-afdbfef81849

📥 Commits

Reviewing files that changed from the base of the PR and between 17a568f and c3818fe.

📒 Files selected for processing (7)
  • .github/workflows/cleanup-staging.yml
  • .github/workflows/deploy-all.yml
  • .github/workflows/deploy-aws-fargate.yml
  • .github/workflows/deploy-aws-lambda.yml
  • .github/workflows/deploy-gcp.yml
  • .github/workflows/destroy-fargate-dev.yml
  • .github/workflows/rollback.yml

📝 Walkthrough

Walkthrough

GitHub Actions workflows now serialize Terraform operations by environment-specific state. The changes cover deployment, cleanup, destroy, and rollback jobs. Automatic Terraform lock deletion was removed from AWS and GCP workflows.

Changes

Terraform state protection

Layer / File(s) Summary
Environment-scoped deployment concurrency
.github/workflows/deploy-aws-fargate.yml, .github/workflows/deploy-aws-lambda.yml, .github/workflows/deploy-gcp.yml, .github/workflows/deploy-all.yml
Deployment jobs now use non-canceling concurrency groups keyed by the resolved environment. Caller workflow comments explain why concurrency remains in called deployment jobs.
Manual Terraform lock recovery
.github/workflows/deploy-aws-lambda.yml, .github/workflows/deploy-gcp.yml, .github/workflows/destroy-fargate-dev.yml
Workflows no longer delete Terraform locks unconditionally. Stranded locks require manual terraform force-unlock recovery.
Cleanup and rollback state coordination
.github/workflows/cleanup-staging.yml, .github/workflows/rollback.yml
Cleanup and rollback jobs now share matching non-canceling concurrency groups for AWS Lambda, AWS Fargate, and GCP Terraform state.

Estimated code review effort: 4 (Complex) | ~45 minutes

Mergeability Score: ⚪ Minimal · up to c3818

This change serializes Terraform state writes by environment and removes unsafe automatic lock deletion while retaining only explicit operator-confirmed recovery. No actionable merge-blocking risk remains beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant DeploymentWorkflow
  participant GitHubActionsConcurrency
  participant TerraformState
  DeploymentWorkflow->>GitHubActionsConcurrency: Resolve deployment environment
  GitHubActionsConcurrency->>TerraformState: Serialize matching state writers
  DeploymentWorkflow->>TerraformState: Apply Terraform changes
Loading

Possibly related issues

Possibly related PRs

  • LeanerCloud/CUDly#1803 — Implements the corresponding environment-based concurrency and manual lock recovery strategy for related workflows.
  • LeanerCloud/CUDly#818 — Also changes Terraform lock handling in deploy-aws-fargate.yml.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR addresses branch-based concurrency and unsafe lock deletion, but leaves the Fargate rollback namespace mismatch unresolved. Use a Fargate-specific rollback concurrency group that matches the Fargate Terraform state namespace, or link this change to the tracked fix.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: environment-based concurrency for AWS and GCP Terraform state writers.
Out of Scope Changes check ✅ Passed The workflow changes and explanatory comments stay within the linked issue objectives.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/1806-serialize-aws-gcp-tfstate

Comment @coderabbitai help to get the list of available commands.

@cristim
cristim merged commit bde563e into main Aug 13, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/m Days impact/internal Team-internal only priority/p1 Next up; this sprint severity/high Significant harm triaged Item has been triaged type/bug Defect urgency/this-sprint Within the current sprint

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Both halves of #1801 are still live on GCP and AWS: branch-keyed concurrency plus unconditional lock deletion

1 participant