[Bug]: flux_bootstrap_git destroy should align with the behavior of flux uninstall

[debaprita]

Describe the bug

Behavior of flux uninstall removes all the CRDs and necessary finalizers from the cluster. However, executing destroy with the Flux Terraform provider causes HelmRelease CRD to get stuck in a Terminating state. It is imperative to run flux uninstall to ensure proper cleanup, including their finalizers. There are helm.toolkit.fluxcd.io/v2beta1 HelmRelease objects defined on the cluster.

Status of HelmRelease CRD after running terraform destroy:

status:
  acceptedNames:
    kind: HelmRelease
    listKind: HelmReleaseList
    plural: helmreleases
    shortNames:
    - hr
    singular: helmrelease
  conditions:
  - lastTransitionTime: "2024-09-29T16:48:09Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2024-09-29T16:48:09Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  - lastTransitionTime: "2024-09-29T16:56:36Z"
    message: CustomResource deletion is in progress
    reason: InstanceDeletionInProgress
    status: "True"
    type: Terminating
  storedVersions:
  - v2

Steps to reproduce

1. terraform apply to install flux on AKS cluster
2. flux installed successfully
3. terrform destroy to uninstall flux
4. kubectl get crds | grep flux
   helmreleases.helm.toolkit.fluxcd.io
5. flux uninstall --namespace=flux-system --keep-namespace
   Are you sure you want to delete Flux and its custom resource definitions: y
   ► deleting components in flux-system namespace
   ► deleting toolkit.fluxcd.io finalizers in all namespaces
   ✔ HelmRelease/ingress-nginx/ingress-nginx finalizers deleted
   ✔ HelmRelease/monitoring/prometheus finalizers deleted
   ► deleting toolkit.fluxcd.io custom resource definitions
   ✗ CustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io deletion failed: customresourcedefinitions.apiextensions.k8s.io "helmreleases.helm.toolkit.fluxcd.io" not found
   ✔ uninstall finished
6. kubectl get crds | grep flux now returns nothing

Expected behavior

Running terraform destroy should have deleted HelmRelease CRD and its finalizers.

Screenshots and recordings

No response

Terraform and provider versions

Terraform v1.9.3 on darwin_arm64
FluxCD 1.30
Kubernetes v2.32.0
Azure RM v3.116.0
AKS Cluster v1.29.4

Terraform provider configurations

provider "flux" {
  kubernetes = {
    host                   = local.kube_provider.host
    client_certificate     = local.kube_provider.client_certificate
    client_key             = local.kube_provider.client_key
    cluster_ca_certificate = local.kube_provider.cluster_ca_certificate
  }
  git = {
    url    = local.url
    branch = var.flux2_git_branch
    ssh = {
      username    = "git"
      private_key = data.azurerm_key_vault_secret.flux2_private_key.value
    }
  }
}

provider "kubernetes" {
  host                   = local.kube_provider.host
  client_certificate     = local.kube_provider.client_certificate
  client_key             = local.kube_provider.client_key
  cluster_ca_certificate = local.kube_provider.cluster_ca_certificate
}

provider "azurerm" {
  features {}
}

locals.tf

locals {
  kube_provider = {
    host                   = coalesce(try(data.azurerm_kubernetes_cluster.this.kube_admin_config[0].host, null), data.azurerm_kubernetes_cluster.this.kube_config[0].host)
    client_certificate     = base64decode(coalesce(try(data.azurerm_kubernetes_cluster.this.kube_admin_config[0].client_certificate, null), data.azurerm_kubernetes_cluster.this.kube_config[0].client_certificate))
    client_key             = base64decode(coalesce(try(data.azurerm_kubernetes_cluster.this.kube_admin_config[0].client_key, null), data.azurerm_kubernetes_cluster.this.kube_config[0].client_key))
    cluster_ca_certificate = base64decode(coalesce(try(data.azurerm_kubernetes_cluster.this.kube_admin_config[0].cluster_ca_certificate, null), data.azurerm_kubernetes_cluster.this.kube_config[0].cluster_ca_certificate))
  }
}

flux_bootstrap_git resource

resource "flux_bootstrap_git" "this" {
    embedded_manifests     = true
    interval               = 10m0s
    version                = "v2.3.0"
    path                   = "./" 
}

Flux version

v2.3.0

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Would you like to implement a fix?

None

[swade1987]

Thanks for submitting the bug. I'll take a look at it this week.

[swade1987]

I wanted to provide a quick update on my availability as a maintainer. I pride myself on transparency and realise my standards have slipped in the last few months.

Since starting a full-time role in September, I have significantly less time to dedicate to this project. I'm now working on issues and pull requests out of hours on a best-effort basis.

Please bear with me if I take longer than usual to respond or review. I remain committed to the project and appreciate your patience and understanding during this transition.

Thank you for your continued support and contributions. I want you to know I'm committed to maintaining better communication.

[aservedio]

Just as an FYI I also hit the same problem, then tried the TF + helm install method and got the same issue.

Flux cli seems to be the only thing that can uninstall and leave cluster in a decent state after, and lets you reinstall flux again after with no issues.

[divStar]

I sadly also hit this issue when I "moved" all files to a different structure within the gitops git repository, that Flux is in charge of. Flux for some reason uninstalled itself - but left helmreleases CRD behind so I couldn't reinstall it until I ran the following command (+ result in the next line):

$ kubectl patch customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io \
  -p '{"metadata":{"finalizers":[]}}' --type=merge
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io patched

It would be nice to not have to do this if somehow possible.