Using Datadog layers and lambda alias causes an error on redeploy

[Aeory]

Expected Behavior

I can redeploy a CDK stack containing lambdas using the Datadog CDK layers without causing an error

Actual Behavior

When redeploying my stack the layer hash changes even if there is no code difference. This results in the CDK trying to create a new version, and causes a Version already exists error and a rollback.

Steps to Reproduce the Problem

1. Create a simple CDK stack

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import {Alias, Code, Function, Runtime} from "aws-cdk-lib/aws-lambda";
import {Datadog} from "datadog-cdk-constructs-v2";
import {Secret} from "aws-cdk-lib/aws-secretsmanager";

export class ReproStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
      const lambda = new Function(
          this,
          "dd-repro",
          {
              functionName: "dd-repro",
              runtime: Runtime.NODEJS_18_X,
              handler: 'index.handler',
              code: Code.fromInline("exports.handler = async (event) => { return 'hello world'; }")
          },
      );

      new Alias(this, `dd-repro-alias`, {
          aliasName: `alias`,
          version: lambda.currentVersion,
          provisionedConcurrentExecutions: 1,
      });

      const ddApiKeySecret = Secret.fromSecretPartialArn(
          this,
          'DatadogApiKeySecret',
          `arn:aws:secretsmanager:${this.region}:${this.account}:secret:YOUR_DATADOG_SECRET_NAME`,
      );

      const dd = new Datadog(this, 'Datadog', {
          nodeLayerVersion: 99,
          extensionLayerVersion: 49,
          addLayers: true,
          site: 'datadoghq.eu',
          apiKeySecret: ddApiKeySecret,
          enableDatadogTracing: true,
          captureLambdaPayload: true,
          enableMergeXrayTraces: true,
          enableDatadogLogs: true,
          injectLogContext: true,
          env: "dev",
          service: 'repro',
          tags: `stage:dev`,
      });

      dd.addLambdaFunctions([lambda])
  }
}

2. Deploy your function.
3. Eventually (I've had to wait a minute or two before) when running cdk diff for your stack will show a new version being made

cdk diff --profile=dev
Stack ReproStack
Resources
[-] AWS::Lambda::Version dd-repro/CurrentVersion ddreproCurrentVersionC5369628d780c301499b40a4b6eef2f7624b39f1 destroy
[+] AWS::Lambda::Version dd-repro/CurrentVersion ddreproCurrentVersionC53696280e70249a63c9be974c8c296a11a167aa
[~] AWS::Lambda::Alias dd-repro-alias ddreproalias187071F3
 └─ [~] FunctionVersion
     └─ [~] .Fn::GetAtt:
         └─ @@ -1,4 +1,4 @@
            [ ] [
            [-]   "ddreproCurrentVersionC5369628d780c301499b40a4b6eef2f7624b39f1",
            [+]   "ddreproCurrentVersionC53696280e70249a63c9be974c8c296a11a167aa",
            [ ]   "Version"
            [ ] ]


✨  Number of stacks with differences: 1

Any subsequent deploys will now fail with this error

Resource handler returned message: "Version already exists: [[...]dd-repro:3]. Please modify the function to create a new vers
ion."

I've found that if you disable the @aws-cdk/aws-lambda:recognizeLayerVersion feature flag then the layer hash becomes consistent and further deploys are totally fine, but ideally we'd be able to leave that bug fix flag on!

The description of the flag makes it sound like one of the properties of the datadog layers is non deterministic?

Specifications

• Datadog Lambda Layer version: 49
• Node version: 99
• CDK version: 2.109.0
• DD construct version: 1.9.0 but I've seen it in 1.8.2 as well.

[astuyve]

Hey @Aeory, thanks for reaching out!

Layers are immutable, so I'm unclear on how that would be non-deterministic

Is the only difference here the lambda version sha?
Are there any other differences in the CF template?
Can you generate the full cdk synth before and after so we could compare?
Would you be able to set enableSourceCodeIntegration: false in your datadog config block and see if the problem persists?

Thanks!

[Aeory]

Sorry for this not replying @astuyve, I completely missed the notification. I'll go through your questions and reply when I can

[Aeory]

Hey @astuyve,

Layers are immutable, so I'm unclear on how that would be non-deterministic

Good question! It seems like people are reporting that the bug fix enabled by this flag can change what is being deployed which could be culprit 🤨

To your questions:

Is the only difference here the lambda version sha?

This is the only difference I observed whilst debugging.

Are there any other differences in the CF template?

Not as far as I'm aware.

Can you generate the full cdk synth before and after so we could compare?

Sure, I've attached them (extension changed to allow upload)

before.txt
after.txt

Would you be able to set enableSourceCodeIntegration: false in your datadog config block and see if the problem persists?

I gave this a go and had the same results as before:

Resources
[-] AWS::Lambda::Version dd-repro/CurrentVersion ddreproCurrentVersionC5369628ba8a80ac2321bbf912ddafaef68593ec destroy
[+] AWS::Lambda::Version dd-repro/CurrentVersion ddreproCurrentVersionC5369628cf76e00b91f042a856f6d86e8bea9b1e 
[~] AWS::Lambda::Alias dd-repro-alias ddreproalias187071F3 
 └─ [~] FunctionVersion
     └─ [~] .Fn::GetAtt:
         └─ @@ -1,4 +1,4 @@
            [ ] [
            [-]   "ddreproCurrentVersionC5369628ba8a80ac2321bbf912ddafaef68593ec",
            [+]   "ddreproCurrentVersionC5369628cf76e00b91f042a856f6d86e8bea9b1e",
            [ ]   "Version"
            [ ] ]
✨  Number of stacks with differences: 1