google_project_service_identity resource does not work when the service account already exists for BigQuery

[JasonSinclair95]

TL;DR

Hi there, I'm having issues using the google_project_service_identity resource to prompt GCP to create the service account bigquery-encryption.iam.gserviceaccount.com for BigQuery KMS encryption on datasets and tables.

The resource hangs at the creation step because GCP has already provisioned the service account. However, on a brand new project, it works as expected.

Expected behavior

I expect the resource to detect when the service account already exists and skip the creation step (or complete successfully), rather than hanging. This would allow us to use this module consistently across both new and existing environments where BigQuery is already integrated.

Observed behavior

The resource hangs for around 20 minutes and eventually fails with a context deadline exceeded error.
I believe this is because the module is applied in an environment where the service account already exists, whereas it works fine in newly provisioned projects.

Terraform Configuration

locals {
  services = [
    "cloudbilling.googleapis.com",
    "billingbudgets.googleapis.com",
    "bigquerydatatransfer.googleapis.com",
  ]
}
resource "google_bigquery_dataset" "this" {
  dataset_id            = var.name
  location              = var.region
  max_time_travel_hours = 168 # set due to plan was always updating resource
  default_encryption_configuration {
    kms_key_name = var.kms_key_id
  }
  labels     = var.labels
  depends_on = [google_kms_crypto_key_iam_member.allow_bigquery_dataset]
}

data "google_project" "project" {}

resource "google_kms_crypto_key_iam_member" "allow_bigquery_dataset" {
  crypto_key_id = var.kms_key_id
  role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
  member        = "serviceAccount:bq-${data.google_project.project.number}@bigquery-encryption.iam.gserviceaccount.com"
  depends_on    = [google_project_service_identity.bq_api]
}

resource "google_project_service_identity" "bq_api" {
  provider   = google-beta
  project    = var.project_id
  service    = "bigquery.googleapis.com"
  depends_on = [google_project_service.enable_api]
}

resource "google_project_service" "bq_data_transfer_api" {
  for_each           = toset(local.services)
  project            = var.project_id
  service            = each.key
  disable_on_destroy = true
}

Terraform Version

terraform {
  required_version = "1.11.0"
}

Terraform Provider Versions

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "6.21.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "6.21.0"
    }
  }
}

Additional information

No response