Skip to content

CustomResource Provider: WaiterStateMachine can't start when stack deployment is retried #35957

@vvigilante

Description

@vvigilante

Describe the bug

By default, the Resource Provider uses the RequestId from Cloudformation to initiate the "waiter" state function. Cloudformation reuses the RequestId when a deployment is retried, and this means that, if the "waiter" started on the failed attempt, it will never succeed on retry, because States will return the error ExecutionAlreadyExists.

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

Retrying a deployment should result in a fresh execution of the "waiter" state function with a new id. This can be achieved by simply omitting the RequestId by default on the StartExecution call.

Current Behavior

Currently, by default we use RequestId from Cloudformation as the name of the state execution, which cannot be repeated for 90 days.

This causes StartExecution to fail with ExecutionAlreadyExists whenever a cloudformation deployment fails after launching the "waiter" and is retried.

Reproduction Steps

  1. Deploy a custom resource so that it succeeds
  2. Update the resource so that the waiter state function runs but fails
  3. Cloudformation will rollback (using a different RequestId), make sure the waiter state function runs again on rollback, and fails again
  4. The stack will be in the ROLLBACK_FAILED state.
  5. Retry the rollback: Cloudformation will use the same RequestId, the "water" will never start again.

Possible Solution

Omit the name on this call or pass a random value or pass a value that is not reused by cloudformation on retries

Additional Information/Context

No response

AWS CDK Library version (aws-cdk-lib)

2.207.0

AWS CDK CLI version

2.1023.0

Node.js Version

20

OS

AL2

Language

TypeScript

Language Version

No response

Other information

No response

Metadata

Metadata

Assignees

Labels

@aws-cdk/custom-resourcesRelated to AWS CDK Custom ResourcesbugThis issue is a bug.effort/mediumMedium work item – several days of effortinvestigatingThis issue is being investigated and/or work is in progress to resolve the issue.p1

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions