Terraform manager is not generating a new .terraform.lock.hcl on provider version updates

[ryancausey]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab w/ ghcr.io/renovatebot/renovate:38.51.1@sha256:8fefd528384e26fd57d59e4e5834392313dd87c71a176d66dbee4fa16238b9c5

Please tell us more about your question or problem

All of my Terraform updates from AWS provider version 5.63.1 to 5.64.0 failed last night because it appears that Renovate Bot did not generate a new .terraform.lock.hcl file. I don't see any errors in the non-debug logs, and there are no notices of an artifact generation failure in the MRs like there normally are if it failed to generate a lockfile.

Did something change with Renovate recently with how it manages Terraform lockfiles to cause this issue?

Logs (if relevant)

Please see the minimal reproduction repository.

[rarkins]

No recent changes to the Terraform manager which would explain that: https://github.com/renovatebot/renovate/commits/main/lib/modules/manager/terraform

Please see if you can reproduce it for us to debug

[secustor]

I've run locally and found that the process to get zip hashes from https://releases.hashicorp.com/terraform-provider-aws/5.64.0/terraform-provider-aws_5.64.0_SHA256SUMS can take quite some time, even though the appears to be under 2KB in size. We should try to identify that and see if it can be fixed if it's unexpected. This could be part of the root cause, causing calculations to fail sometimes.

The hashes form the _SHA256SUMS files are the ones for the archive not the one for the contents and the later is what Terraform is using to generate the h1 hashes an that is what takes so long.
This is as Renovate has to download the archives for all arches to generate the hashes.
The h1 hashes generated they should be cached pretty much indefinitely.

Yeah, seems like this would happen if the Terraform registry returns empty builds, though this only cached for default TTL and not indefinitely.

If here an empty array is returned:

renovate/lib/modules/manager/terraform/lockfile/index.ts

Lines 201 to 205 in d1af677

	(await TerraformProviderHash.createHashes(
	registryUrl,
	updateLock.packageName,
	newVersion!,
	)) ?? /* istanbul ignore next: needs test */ [],

Then no updates are written to the lock file:

renovate/lib/modules/manager/terraform/lockfile/index.ts

Lines 211 to 217 in d1af677

	// if no updates have been found or there are failed hashes abort
	if (
	updates.length === 0 ||
	updates.some((value) => !value.newHashes?.length)
	) {
	return null;
	}

Fetching builds:

renovate/lib/modules/manager/terraform/lockfile/hash.ts

Lines 157 to 164 in 5f7d22e

	const builds = await TerraformProviderHash.terraformDatasource.getBuilds(
	registryURL,
	repository,
	version,
	);
	if (!builds) {
	return null;
	}

So this should resolve itself if the default TTL of the App cache is reached 🤔

[ryancausey]

What is the default TTL? Is there a default TTL for a self hosted renovate bot (GitLab)?

[secustor]

The general default TTL is 30min though this can be overwritten via code or cacheTtlOverride

[ryancausey]

I just re-ran it again using the "retry mr" checkbox and it still did not update the lockfile. In the reproduction repo linked I didn't make any changes to the renovate.json file, so it should be using all the default settings. Why wasn't the lockfile regenerated still? It's definitely been more than 30 minutes.

Completed at: Aug 29, 2024, 12:10:01

DEBUG: updateArtifacts for updatedPackageFiles (branch="renovate/aws-5.x")
DEBUG: terraform.updateArtifacts(main.tf) (branch="renovate/aws-5.x")
DEBUG: Updating constraint "5.63.1" to replace "5.63.1" with "5.64.0" for "hashicorp/aws" (branch="renovate/aws-5.x")
DEBUG: Updated 1 package files (branch="renovate/aws-5.x")
DEBUG: No updated lock files in branch (branch="renovate/aws-5.x")
DEBUG: isBranchConflicted(main, renovate/aws-5.x) (branch="renovate/aws-5.x")
DEBUG: branch.isConflicted(): using cached result "false" (branch="renovate/aws-5.x")
DEBUG: 1 file(s) to commit (branch="renovate/aws-5.x")
DEBUG: Preparing files for committing to branch renovate/aws-5.x (branch="renovate/aws-5.x")
DEBUG: Setting git author name: renovate[bot] (branch="renovate/aws-5.x")
DEBUG: Setting git author email: 29139614+renovate[bot]@users.noreply.github.com (branch="renovate/aws-5.x")

[nipr-jdoenges]

Are you sure the hash cache TTL isn't a week?

renovate/lib/modules/manager/terraform/lockfile/hash.ts

Line 23 in 5f7d22e

static hashCacheTTL = 10080; // in minutes == 1 week

[tom-256]

We have same issue.
~> 5.40.0 -> ~> 5.63.1 updated .terraform.lock.hcl
but
~> 5.63.1 -> ~> 5.64.0 did not update .terraform.lock.hcl

Is there any additional information you need for debugging or reproducing the issue?

[nipr-jdoenges]

I will note we are seeing this in our own projects
(GitLab w/ renovate/renovate:38.27.0-full@sha256:6fe495fb0a43e5370bf81df61ee80df334956dd53dc738b812848b07bc091e5f)

And the problem updates all appear to be
aws provider ~> 5.64.0, with previous proposals for upgrades up to and including ~> 5.63.1 all appearing to correctly include the hash updates in lockfiles.

Very well may be a coincidence, but it does seem to be related in some way to the 5.64.0 release of the aws terraform provider.

[nipr-jdoenges]

Possibly related? hashicorp/terraform-provider-aws#39004

[rarkins]

Very good chance it's related!

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

Please follow these steps:

1. Read our guide on creating a minimal reproduction.
2. Go to our minimal reproduction template repository.
3. Select the Use this template button to create a new repository based on the template.
4. Work on the minimal reproduction in your own repository.
5. Fill out the information in your repository's README.md.
6. Add the link to your reproduction to the first post of your Discussion. If you are not the original author, you can post a new comment with the link.

Good luck,

The Renovate team

[mikutas]

We had the same issue, but after 5.65.0 of aws is released the issue is resolved

[ryancausey]

I can confirm this appears to have resolved it. I'm going to assume it has to do with the Terraform release issue pointed out here: #30984 (reply in thread)

[mikutas]

seems reoccurred in 5.90.1 hashicorp/terraform-provider-aws#41763

[mikutas]

fixed