pip-compile update breaks if dependency is grouped with custom regex manager updates

[chludwig-haufe]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab, Renovate 39.158.1 and 38.142.7

Please tell us more about your question or problem

Minimal reproduction repository: https://github.com/chludwig-haufe/renovate-pip-compile

I manage the versions of some Python packages I install in a Docker base image with pip-compile. The README of this project lists the versions of the most relevant packages. These versions are stored in AsciiDoc document properties and updated using Renovate's custom regex manager.

As long as a package is not included in the README, the pip-compile manager generates an MR that icludes updates to both requirements.in and requirements.txt (as expected).

However, if a package is also included in the README (i.e., there is a comment for this dependency matched by the regex manager), then the generated MR contains the changes to README.adoc and requirements.in only. The update of requirements.txt that must come together with the change to requirements.in is lost.

I see in the debug logs that the requirements.txt is generated. But I didn't find any messages that would explain why the generated file is not carried over into the MR.

Logs (if relevant)

No response

[rarkins]

If it only applies to grouping with the custom manager then please update the title to match.

[chludwig-haufe]

Now it get's confusing!

In order to test whether this happens with other managers as well, I added a file req.txt (with the same packages as in the file managed by pip-compile) and configured the pip_requirements manager to update req.txt. And suddenly the generated MRs contained updates to all four files: The new req.txt, README.adoc, requirements.in, and requirements.txt.

Then I disabled the custom manager: The results of the pip-compile manager and the pip_requirements manager were grouped together without problems. So: On first glance, yes, this seems specific to the custom regex manager.

[chludwig-haufe]

I think I found the cause of this issue. It has nothing to do with the pip-compile or the custom.regex managers per se; it's just the combination of managers that use lock files with others that don't. The lockFiles attribute is not reliably copied into the branch config.

Consider generateBranchConfig() in lib/workers/repository/updates/generate.ts:

• Within the loop for (const branchUpgrade of branchUpgrades) (lines 255ff), branchUpgrade.lockFiles is defined (and a non-empty list) if the corresponding manager uses a lock file. In my example:
  • If branchUpgrade refers to the upgrade of, say, the wheel dependency found by the pip-compile manager, then branchUpgrade.packageFile contains the name of the requirements.in file and branchUpgrade.lockFiles contains the name of the corresponding requirements.txt file.
  • If branchUpgrade refers to the upgrade of the same wheel dependency found by the regexp manager in the README file, then branchUpgrade.packageFile contains the name of the README file and branchUpgrade.lockFiles is undefined.
• In line 331, the upgrade config objects are sorted.
• In line 358, the head of the result list is expanded into the config object. Now config may or may not have a lockFiles attribute. None of the remaining code in generateBranchConfig() copies the lockFiles information.

The branch config is eventually passed into patchConfigForArtifactsUpdate() in get-updated.ts. This function is called for each package file in the branch config (a.o. from line 339 of the function getUpdatedPackagesFile() in the same file). However, if the config object in the previous bullet point ended up without a lockFiles attribute, then the condition is.nonEmptyArray(updatedConfig.lockFiles) is not met and patchConfigForArtifactsUpdate() returns the config unmodified. In consequence, the lock files are not added to the branch.

I don't know the best approach for a proper fix:

• Similar to the way constraints are collected in generateBranchConfig(), the list of lock files could be placed into the branch config. This could at least serve as a signal that some lock files should be added to the branch.
• I don't get the point of the condition in patchConfigForArtifactsUpdate(). According to a comment, the function is a workaround anyway. But the referenced PR didn't tell me what could break if the condition is.nonEmptyArray(updatedConfig.lockFiles) would be dropped.

[chludwig-haufe]

As said, I don't really understand PR #27319 that introduced patchConfigForArtifactsUpdate(). Maybe @not7cd can shed some light on the idea behind it?

[not7cd]

IIRC, I did this workaround so all lock files would be handled by artifacts update. For some reason only one (first) would be passed and handled.

[not7cd]

I think it was a limitation at the time that one package file could have one lock file. Or at least one lock file would be handled per package file in artifacts update function. But at this point I don't remember well.

[not7cd]

I will look into it later in the weekend. I honestly don't remember.

[chludwig-haufe]

I created PR #34701.

[not7cd]

I see in the debug logs that the requirements.txt is generated. But I didn't find any messages that would explain why the generated file is not carried over into the MR.

Would you like to provide these logs?

[chludwig-haufe]

The output of docker container run -e LOG_LEVEL=debug --env-file ./renovate-github.env --rm ghcr.io/renovatebot/renovate:39.170.1 --platform=github --detect-host-rules-from-env=true --dry-run=full --schedule '["at any time"]' chludwig-haufe/renovate-pip-compile 2>&1 | nl | tee pip-compile.log
is attached.

It seems I was wrong regarding the statement you quoted: Line 773f mentions an execution of pip-compile for aniso8601. But there are no corresponding calls to pip-compile for wheel or httpx.

This matches the results of my (later) debug session: For the latter two packages, updatedConfig in patchConfigForArtifactsUpdate() has no attribute lockFiles. Therefore, packageFile.lockFiles is not copied.

What is the point of the condition is.nonEmptyArray(updatedConfig.lockFiles)? If a packageFilein getUpdatedPackageFiles() has a lock file, why should this lock file not be updated in the generated branch?

[not7cd]

graph TD
       requirements.txt[[requirements.txt]]
       requirements.in --> requirements.txt 

Loading

(repository=chludwig-haufe/renovate-pip-compile, baseBranch=main)
for reference, theses are detected files by pip-compile manager