Shared/Global lockfiles management - turborepo prune "equivalent"?

[castarco]

Hello,

I'm quite new in the monorepos landscape, and I didn't use Nx yet (I like to master more basic tools before jumping into more powerful ones, to avoid situations where half my stack feels like black magic).

One of the problems I faced is ensuring that I can reuse docker builder cache layers as most as possible in the presence of "shared/global" lockfiles (as it happens when using yarn workspaces, where there's a single global yarn.lock file). As you might already know/imagine, this is a problem because some parts of the lockfile can change without affecting the real behavior of the build, but because its checksum changes, the cache layer associated to it has to be invalidated.

Because of this, I wrote a small (experimental) tool: https://github.com/Coder-Spirit/lock-distill , that takes care of creating "distilled" lockfiles that can be mounted/copied during the build process, minimising the amount of caches invalidation.

I discussed this with a colleague, and he pointed out the existence of the prune command in turborepo ( https://turborepo.org/docs/reference/command-line-reference#--docker ), that does basically the "same"... but at a much greater scale.

How does Nx and its docker-related plugins deal with this case?

[imolorhe]

Is there any chance someone can respond to this?

[castarco]

I've been working with Nx for a project of my company, and it seems that in Nx case, it goes farther than having a single lockfile, it also works with a single package.json file... which makes things much, much worse.

The feedback I got time ago from the team is that they plainly didn't care about that use case, diminishing its importance. It was not even a "polite no for X reasons" what I got from them, but more like a "what you say is irrelevant".

[imolorhe]

I ended up including turbo in my setup just for this feature 😕 so now I use both turbo and nx, along with the native yarn workspaces

[thardy]

I've done the same. I added turbo specifically for the ability to cleanly create docker builds in my monorepo with local shared packages. The lack of support for building concise docker images from monorepo tools like Nx is pathetic. Or maybe it's a lack of documentation and guidance. If there's a way to do it with Nx, I simply could not find it after days of searching.

[jdefting]

I wonder if this section of the docs addresses this issue https://nx.dev/recipes/ci/ci-deployment#prepare-applications-for-deployment-via-ci? Executors like @nx/js:tsc already generatea pruned package.json and have a generateLockfile flag which generates a pruned lockfile.

[castarco]

Generating pruned lockfiles would make sense. But generating pruned package.json files is certainly not a good idea, as it makes it very difficult (almost impossible) to keep proper isolation between packages (some of them happen to represent services) in the monorepo.

The only reason I can imagine to justify that is to keep all dependencies updated in step lock, always at the same version for every package inside the monorepo (which is a "sensible approach", but is not strictly necessary to ensure stability and maintainability, and it's not free from subjective bias). More on this: where it makes most sense is in frontend development, but it provides almost no value (or even negative value) in microservices development, except for very specific libs related to inter-service communication.

Having said that, even if we wanted to keep all our dependencies updated at step lock, having a single package.json file is not the only option to achieve that goal (and I'd say that's one of the worst possible ones). We could have very a very simple script/tool that runs as a git hook, or in CI, to verify that dependencies' versions are properly synchronised.