@marcusricardoaguiar changing on two fields, name and project, trigger replacement. It works fine with me after I renamed the resource.

What did you do to trigger the recreation?

After making some changes on the resource google_compute_security_policy that triggers the resource recreation,

@marcusricardoaguiar changing on two fields, name and project, trigger replacement. It works fine with me after I renamed the resource.

What did you do to trigger the recreation?

After making some changes on the resource google_compute_security_policy that triggers the resource recreation,

Hey @edwardmedia , for instance, the last time it happened, it was a rule that I replaced. I changed the rule priority and the expressions. Then, it triggered the security policy recreation.

@marcusricardoaguiar changing on two fields, name and project, trigger replacement. It works fine with me after I renamed the resource.

What did you do to trigger the recreation?

After making some changes on the resource google_compute_security_policy that triggers the resource recreation,

However, I haven't changed the project nor the name.

@marcusricardoaguiar can you detail the steps to repro, specific the way to trigger recreation? Besides project and name, I do not see other fields that will trigger it.

@marcusricardoaguiar can you detail the steps to repro, specific the way to trigger recreation? Besides project and name, I do not see other fields that will trigger it.

HI @edwardmedia , I've just found out what's going on. I've just created a google_compute_security_policy and also attached a backend service to this security policy as shown below:

resource "google_compute_security_policy" "cloud_armor" {
  name = "cloud-armor"
  provider = google-beta

  rule {
    action   = "deny(403)"
    description = "Deny violations"
    priority = 4
    match {
      expr {
        expression = "evaluatePreconfiguredExpr('*****')"
      }
    }
  }

  rule {
    action   = "deny(403)"
    description = "Block all traffic"
    priority = 2
    match {
      expr {
        expression = join(" ", [
          "origin.region_code == 'BR'"
        ])
      }
    }
  }

  # Every security policy must have a rule with priority 2147483647 and match "*".
  rule {
    action   = "allow"
    priority = "2147483647"
    match {
      versioned_expr = "SRC_IPS_V1"
      config {
        src_ip_ranges = ["*"]
      }
    }
    description = "Default rule"
  }
}

# PgAdmin Internal Service
resource "google_compute_backend_service" "pgadmin_cloudrun" {
  name                            = "pgadmin-service"
  enable_cdn                      = false
  port_name                       = "https"
  timeout_sec                     = 90
  connection_draining_timeout_sec = 90
  protocol                        = "HTTPS"

  backend {
    group = module.pgadmin.pgadmin4_instance_group
  }

  health_checks = [
    google_compute_https_health_check.pgadmin4_health.id,
  ]

  log_config {
    enable = true
  }

  security_policy = google_compute_security_policy.cloud_armor.self_link
}

Then, I tried to update the security policy and I added a new rule with a high priority:

resource "google_compute_security_policy" "cloud_armor" {
  name = "cloud-armor"
  ...
  rule {
    description = "Login rate limit"
    action      = "throttle"
    priority    = 28
    match {
      expr {
        expression = join(" ", [
          "request.method == 'POST'",
          "&& !has(request.headers['*****'])"
        ])
      }
    }

    rate_limit_options {
      conform_action = "allow"
      exceed_action = "deny(429)"

      enforce_on_key = ""

      enforce_on_key_configs {
        enforce_on_key_type = "HTTP_HEADER"
        enforce_on_key_name = "*******"
      }

      rate_limit_threshold {
        count = 5
        interval_sec = 60
      }
    }
  }
  ...
}

In this case, since this security policy has been used by the backend service, it will try to create the new one before deleting it. Then, it crashes because of name already exist (409 error).

@marcusricardoaguiar Using below config, I tested by 1) initial apply and 2) uncomment the 3rd rule and apply, and did not hit the problem. Adding the 3rd rule did not trigger recreate google_compute_security_policy. Can you try my config? Also what version of the provider do you use?

resource "google_compute_security_policy" "policy" {
  name = "issue14738-1"
  //type = "CLOUD_ARMOR"
  type = "CLOUD_ARMOR_INTERNAL_SERVICE"

  rule {
    action   = "deny(403)"
    priority = "1000"
    match {
      versioned_expr = "SRC_IPS_V1"
      config {
        src_ip_ranges = ["9.9.9.0/24"]
      }
    }
    description = "Deny access to IPs in 9.9.9.0/24"
  }

  rule {
    action   = "allow"
    priority = "2147483647"
    match {
      versioned_expr = "SRC_IPS_V1"
      config {
        src_ip_ranges = ["*"]
      }
    }
    description = "default rule"
  }
/*  uncomment for the 2nd apply
  rule {
    description = "Login rate limit"
    action      = "throttle"
    priority    = 28
    match {
      expr {
        expression = join(" ", [
          "request.method == 'POST'",
          "&& !has(request.headers['*****'])"
        ])
      }
    }

    rate_limit_options {
      conform_action = "allow"
      exceed_action = "deny(429)"

      enforce_on_key = ""

      rate_limit_threshold {
        count = 5
        interval_sec = 60
      }
    }
  }
  */
}


resource "google_compute_backend_service" "default" {
  name          = "backend-service"
  health_checks = [google_compute_http_health_check.default.id]

  security_policy = google_compute_security_policy.policy.self_link
}

resource "google_compute_http_health_check" "default" {
  name               = "health-check"
  request_path       = "/"
  check_interval_sec = 1
  timeout_sec        = 1
}

@marcusricardoaguiar Using below config, I tested by 1) initial apply and 2) uncomment the 3rd rule and apply, and did not hit the problem. Adding the 3rd rule did not trigger recreate google_compute_security_policy. Can you try my config? Also what version of the provider do you use?

resource "google_compute_security_policy" "policy" {
  name = "issue14738-1"
  //type = "CLOUD_ARMOR"
  type = "CLOUD_ARMOR_INTERNAL_SERVICE"

  rule {
    action   = "deny(403)"
    priority = "1000"
    match {
      versioned_expr = "SRC_IPS_V1"
      config {
        src_ip_ranges = ["9.9.9.0/24"]
      }
    }
    description = "Deny access to IPs in 9.9.9.0/24"
  }

  rule {
    action   = "allow"
    priority = "2147483647"
    match {
      versioned_expr = "SRC_IPS_V1"
      config {
        src_ip_ranges = ["*"]
      }
    }
    description = "default rule"
  }
/*  uncomment for the 2nd apply
  rule {
    description = "Login rate limit"
    action      = "throttle"
    priority    = 28
    match {
      expr {
        expression = join(" ", [
          "request.method == 'POST'",
          "&& !has(request.headers['*****'])"
        ])
      }
    }

    rate_limit_options {
      conform_action = "allow"
      exceed_action = "deny(429)"

      enforce_on_key = ""

      rate_limit_threshold {
        count = 5
        interval_sec = 60
      }
    }
  }
  */
}


resource "google_compute_backend_service" "default" {
  name          = "backend-service"
  health_checks = [google_compute_http_health_check.default.id]

  security_policy = google_compute_security_policy.policy.self_link
}

resource "google_compute_http_health_check" "default" {
  name               = "health-check"
  request_path       = "/"
  check_interval_sec = 1
  timeout_sec        = 1
}

Hi @edwardmedia , I've been using google-beta provider version 4.66.0. I've tried your suggestion and it worked as you mentioned. However, if we include the block enforce_on_key_configs it will trigger the policy recreation. Instead of applying your suggested new rule, I've just tried to add the rule below and it triggered the policy recreation:

  rule {
    description = "Login rate limit"
    action      = "throttle"
    priority    = 28
    match {
      expr {
        expression = join(" ", [
          "request.method == 'POST'",
          "&& !has(request.headers['******'])"
        ])
      }
    }

    rate_limit_options {
      conform_action = "allow"
      exceed_action = "deny(429)"

      enforce_on_key = ""

      enforce_on_key_configs {
        enforce_on_key_type = "HTTP_HEADER"
        enforce_on_key_name = "****"
      }

      rate_limit_threshold {
        count = 5
        interval_sec = 60
      }
    }
  }

@marcusricardoaguiar oh, yes, that is right. enforce_on_key_configs is only available in beta and it is a field that could trigger recreation.

Ah I see, thanks @edwardmedia , so, do you know if this will not trigger recreation once it's not beta anymore?
Thanks!

@marcusricardoaguiar any changes on that field will trigger recreation once it is GA. I do not have a solution off the top of my head to your situation.

@ryanyuan what do you think?

hey @edwardmedia, I reckon you wanted to tag @roaks3

I've checked through the code, and nothing stood out as an obvious cause of the 409. The force_replacement behavior will call the resource's delete function and then its create function, which both appear to properly retrieve operations and wait for them to complete.

My suspicion at this point is that the resource is deleted and allowed to proceed to the create, but the resource's name is not yet available (google_storage_bucket has similar behavior where a name cannot be reused immediately after a bucket is deleted, and it caused problems with ForceNew: GoogleCloudPlatform/magic-modules#7314).

I could see the main potential issue here is:
google_compute_backend_service has the security_policy attribute which is string instead of an object type of google_compute_security_policy ;

We are creating two separate resources here and mapping it after the resources are created. This will have an issue because the proper dependency is missing.

b/297932215

@santoshpandit1 even adding dependency between LB BE to Security policy didn't help. Is there any resolution? I was updating a match part

Hello there, I just checked this has been available in beta for some time now so I am bumping it to GA.

Additionally they removed the force_new flag for the enforce_on_key_configs block so now it updates-in-place instead of recreate.

Yea, looks like enforce_on_key_configs was fixed in GoogleCloudPlatform/magic-modules#8165. That leaves only name and project that can trigger replacements, for which we were not able to observe this issue.

@maxi-cit we should be able to close with your change (or even close it now), and then let users raise a new issue if there are still problems with the replacement.