Add Manager for Cruft

[maxnitze]

Tell us more.

Hey,

Cruft is a tool that lets you create new repositories from templates (e.g. cookiecutter templates) and keepthem in sync when the template repository changes.

https://cruft.github.io/cruft/

Do you think this would be a good addition as a manager in RenovateBot? Was it already discussed maybe? (I couldn't find anything in this repo.)

If you think this would be a nice addition, I would gladly try my best to implement it. If someone could point me to some documentation on how to start :)

[github-actions[bot]]

Hi there,

You're asking us to support a new package manager. We need to know some basic information about this package manager first. Please copy/paste the new package manager questionnaire, and fill it out in full.

Once the questionnaire is filled out we'll decide if we want to support this new manager.

Good luck,

The Renovate team

[maxnitze]

New package manager questionnaire

Did you read our documentation on adding a package manager?

• I've read the adding a package manager documentation.

Basics

Name of package manager

cruft

What language does this support?

cruft/cookiecutter

How popular is this package manager?

• active development
• 36 contributors
• ~1k stars on GitHub

Does this language have other (competing?) package managers?

• Yes (give names)
• No

Not, that I know of.

---

Package File Detection

What type of package files and names does it use?

.cruft.json

What fileMatch pattern(s) should be used?

hardcoded filename .cruft.json

Is it likely that many users would need to extend this pattern for custom file names?

• Yes
• No

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

No

---

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

No

Is there a reason why package files need to be parsed together (in serial) instead of independently?

No

What format/syntax is the package file in?

• JSON
• TOML
• YAML
• Custom (explain below)

How do you suggest parsing the file?

• Off the shelf parser
• Using regex
• Custom-parsed line by line
• Other

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, development dependencies, etc?

• Yes, production and development dependencies
• No, all dependencies are treated the same

List all the sources/syntaxes of dependencies that can be extracted

There is always only one dependency, that is a Commit in another (Git) repository.

Describe which types of dependencies above are supported and which will be implemented in future

• (Git) repository only

---

Versioning

What versioning scheme does the package file(s) use?

Git commit hash

Does this versioning scheme support range constraints, e.g. ^1.0.0 or 1.x?

• Supports range constraints (e.g ^1.0.0 or 1.x)
• No

---

Lookup

Is a new datasource required? Provide details

• Yes, provide details.
• No.

There is git-ref already. Maybe that one can be used here?

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

Git repositories are configured at key template in the config file. Those can be private repositories too.

Do the package files have any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc.) that should be used in the lookup procedure?

No.

Will users need the ability to configure language or other constraints using Renovate config?

No.

---

Artifacts

Are lock files or checksum files used? Are they mandatory?

Not exactly a lock file, but the commit hash in the config file is changed when the repo is updated from the template.

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

Binary is called cruft and can be installed from pip: pip3 install cruft

To update a repository from its template the command cruft update is used. cruft check to check whether an update is available.

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or environment variables? Do you recommend the cache be kept or disabled/ignored?

No cache.

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

There is only the config file.

Other

Is there anything else to know about this package manager?

It also needs cookiecutter installed to update from a template.

https://www.cookiecutter.io/

[maxnitze]

Yep, that is exactly how it is done. cruft update will update the repository and - if successful - update the commit hash in the .cruft.json file.

Here's an example of the config file

{
  "template": "https://github.com/my-org/my-repo.git",
  "commit": "6aa48f529bd8bd0c13c428f8c948dae811c7c015",
  "checkout": null,
  "context": {
    "cookiecutter": {
      "project_name": "team-dev-platform-and-infra-docs",
      "repo_host": "github.com",
      "repo_owner": "my-org",
      "repo_slug": "my-repo",
      "package": "com.example",
      "_template": "https://github.com/my-org/my-repo.git"
    }
  },
  "directory": null
}

Every key in context.cookiecutter is defined by the cookiecutter template in the template repository. The values are what the user set when creating the repository from the template. So they are not important for any up-to-date check.

I had a typo in my post: The command to check for an update is cruft check. I fixed it in the questionnaire. But in general it will only do something like "is there a newer commit on the default branch of the template repo than the one in commit". So the cruft binary is not needed for it.

[rarkins]

Will cruft update also update the .cruft.json file? Because we don't want that, what we're looking for is a way for it to sync the hash file to the new .cruft.json we generate.

[maxnitze]

Hm. I don't quite understand what the workflow is here.

Let me try to explain what cruft is doing: Cruft calculates the diff of the current state of the repository (given by the commit hash in the .cruft.json file) with the most recent commit of the template repository using the context given. It then applies a patch of this diff to the current repository and updates the commit hash in the .cruft.json file.

This is all done by cruft update. There are some more commands, like cruft check, which just checks whether there is a new commit to update to. Or cruft diff, that prints the diff, between the diff between the "downstream repository" and the state after an update was done.

Does this somehow fit into the RenovateBot workflow?

[rarkins]

Could you create a demo repo? Ideally it contains a few dependencies, at least one of which is out of date. Then manually create a PR updating one of them in the way you'd like Renovate to do.

[maxnitze]

There is only one dependency (by design). But I can try to create one, yeah.

[github-actions[bot]]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[maxnitze]

I created a minimal setup to show what an update would look like.

It consists of the following three repositories:

• The template Repository https://github.com/maxnitze/cruft-example-cookiecutter-template
• A repository with the vanilla generated content: https://github.com/maxnitze/cruft-example-my-project-plain
• A repository with changed content, that leads to a merge conflict: https://github.com/maxnitze/cruft-example-my-project-conflict

In the following I'll explain how I created the repositories and the PRs inside of them.

The Setup

First I created the template repository. Back then the HEAD pointed to commit 5536d4fdb58ddb5c9b1dcf1eff05cc58ed2ddc38 - initial cookiecutter template.

I then created the two downstream projects by doing the following (example for the plain repo):

• generate a new project from the template

  cruft create \
      --no-input \
      --extra-context '{ "project_name": "my-project-plain", "project_display_name": "My Project Plain" }' \
      [email protected]:maxnitze/cruft-example-cookiecutter-template.git

• create the GitHub repo "cruft-example-my-project-plain" and push the project to it

  cd cruft-example-my-project-plain/
  git init
  git add .
  git commit -am 'Initial generated project'
  git branch -M main
  git remote add origin [email protected]:maxnitze/cruft-example-my-project-plain.git
  git push -u origin main

  • the same for "cruft-example-my-project-conflict"
• I then introduced a change in the "conflict" project, in which I replaced the Git URL in the README.md with the https URL to simulate a change, that will result in a conflict later on.
• Lastly I updated the template repository with a new commit, that wraps the Git URL in quotation marks.

The Update

❗ this is where RenovateBot would get in

• I created a new branch renovate/cruft-update in both downstream repositories.
• I executed cruft update -y in both projects. This now detected, that there was an update in the template repository and patches the projects with the changes from the template.
  • In the "plain" repository this just works as there was no (breaking) change since the creation.
  • In the "conflict" repository this leads to a merge conflict in the README.md.
• I then pushed the changes in both repositories and created PRs:
  • cruft update maxnitze/cruft-example-my-project-plain#1
  • cruft update with conflicts maxnitze/cruft-example-my-project-conflict#1

[maxnitze]

Yep. It checks what commit is set in the config file and calculates the diff between that and whatever is given in the --checkout parameter (default the repo HEAD).

[rarkins]

In that case we'll need the getArtifacts() function for cruft manager to reset the hash back to what it was before running the cruft CLI

[maxnitze]

Is it even necessary to manually update the commit hash in the first place?

[rarkins]

Yes for now, because that's how every other manager works

[jackmac92]

Not an expert with either cruft or renovate, but found this issue because I think an integration would be very useful. Thanks for taking the time to setup a demo @maxnitze !!

Is the current blocker mainly on renovate overriding how the commit hash is written? Are there other/bigger concerns?

Or @rarkins is there some way to achieve this with a custom regex manager?

[github-actions[bot]]

Hi there,

This is intended as a polite, automated request that users avoid @ mentioning repository maintainers like @rarkins or @viceice. Doing so causes annoying mobile notifications and makes it harder to maintain this repository.

We know it might be common elsewhere but we participate in hundreds of discussions a week and would need to turn off GitHub mobile notifications if we were mentioned in every one.

As a general rule, we will read and respond to all discussions in this repository, so there is no need to mention us.

Thanks, the Renovate team