Module does not work with Ephemeral Agents/Runners

[dansiviter]

This module is not designed for ephemeral runners/agents such as Terraform Cloud Remote Operations that cannot have gcloud on the PATH and cannot have custom base images. If anything fails you have to unwind and re-apply all the changes again to force gcloud to be re-downloaded.

Example:

module.asm_install.module.gcloud_kubectl.null_resource.additional_components[0] (local-exec): Executing; ["/bin/sh" "-c" ".terraform/modules/asm_install/scripts/check_components.sh .terraform/modules/asm_isntall/cache/xxxx/google-cloud-skd/bin/gcloud kubectl,kpt,beta.kustomize"]

Error: Error running command '.terraform/modules/asm_install/scripts/check_components.sh .terraform/modules/asm_isntall/cache/xxxx/google-cloud-skd/bin/gcloud kubectl,kpt,beta.kustomize': exit status 127. Output:

In the above example, due to a previously failed run the repeat run then fails to get passed this as it assumes gcloud exists in the cache. However, when it runs it returns a 127 command not found code.

[morgante]

Are you setting the GCLOUD_TF_DOWNLOAD variable? it is specifically to enable downloading in environments where gcloud is not present.

[dansiviter]

Apologies, I should have stated skip_download: false is set.

[morgante]

I see, I'm not actually sure how we can fix this though. Open to ideas.

[dansiviter]

I would argue we need to break the link between persisted TF state and local caching. That would mean dropping the following:

• random_id.cache,
• null_resource.prepare_cache,
• null_resource.download_gcloud,
• null_resource.download_jq.

Then performing caching from within each script to check for gcloud (or jq) each time. This will maintain the exiting functionality running locally and caching the binaries. When this runs in an ephemeral agent/runner it will be slower as it'll be downloading each time but should at least it should complete (bar any download issues).

[morgante]

Moving the logic into each script (and, therefore, into bash) would probably work, though it does increase complexity.

However, I should be clear it's not necessarily a top priority for our team. In general we've found adding more functionality for downloading gcloud to add complexity (and a source of bugs) and are focusing more on having it available in all execution environments. So we're probably unlikely to work on this soon.

If this is a priority for you, I'd be happy to review a pull request though. I do think it's reasonable to move the check into each script.

[dansiviter]

If this helps focus the issue, we hit this when trying to install ASM (via terraform-google-kubernetes-engine/modules/asm) so if there is a way in the works to do that without this module (via kubectl-wrapper) then it will be less of an issue for us.

[morgante]

We'd love to move ASM install away from a bash script, but it definitely won't be happening soon.

[AndreaGiardini]

I am hitting the same problem on terraform cloud. Have you found any workaround?

[onsails]

I've managed to work-around adding for_each to the module arguments:

module "gcloud" {
  # force new state on each apply
  for_each = {
    timestamp = "${timestamp()}"
  }

  ...
}

Of course, it works for idempotent create_cmd and special handling in delete_cmd