In the future release, deps does not load package dependencies transitively

[AllanBlanchard]

For the Frama-C team.

In our tests, we use dune rules to execute our tests. We have a special format that list direct dependencies of the tests so that we list the packages that are used by the test and we expected deps to load the dependencies transitively. From this commit:

95da007

deps does not load dependencies anymore and this is a big problem for us, since we now have to list all dependencies by hand, which means that if our components slightly change we have to update dependencies everywhere. Or we could eventually build a dependency walker ... while dune has all necessary information.

Is there a way for us to keep the previous behavior? If not, is there a workaround?

[Alizter]

This behaviour is indeed intentional and there doesn't seem to be a good way of reliably using the old behaviour.

The crux of the issue is that the "transitive closure" that appeared to be the case before was an unintentional side-effect of the install staging area that was being used for package dependencies.

In dune there are 3 kinds of packages: Lock directory packages, workspace packages and external packages. The (package) dependency should be agnostic to whichever package is there, otherwise users would observe different semantics depending on their setup.

For lock directory and workspace packages, the case of transitive package dependencies makes perfect sense. However for external packages, we no longer have the dependency information. The external packages can be further subdivided into dune installed packages and findlib packages.

We don't currently include package dependency metadata in dune-package, but if we could then we would have a more complete package dependency tree. However, findlib packages still fall outside of these semantics and there is no reliable way to get the package dependencies of a findlib pacakge from what I can tell.

... while dune has all necessary information.

This part is not true. Consider several packages in a workspace, like you have for Frama-C. If we had some sort of package closure mechanism, dune would be able to easily create one for said workspace packages.

Now consider how opam would install these packages individually. They would be built with dune build -p ... so wouldn't have access to the other packages' contents due to the package mask, therefore they would only have access to external packages in this view. Any build rules or tests that mark the closure of a package dependency would no longer work the same.

In dune we want to keep the behaviour homogeneous amongst packages so that users specifying rules don't have to think about packaging details. The current package deps are precise and conservative, and more importantly have correct consistent semantics in differing situations.

The solution here is to write out the package deps that you actually need explicitly. If you require extra common dependencies as packages, then your dep walker should take those into account.

[zilbuz]

I'm not sure I understand, sorry if my remarks or questions are a bit
naive.

You are saying that the transitive closure was an unintentional
side-effect of the install staging area. But this was only the case for
workspace packages right? In previous versions of dune, if you were
using a (package) dependency with an external package, then that
external package and its transitive dependencies had to be available on
your system for dune to pick them up and be able to build your package,
but it did not copy them in the install staging area?

I don't really understand why dune wouldn't follow transitive
worskpace dependencies of workspace packages since in that case it
should have all necessary information. I don't expect dune to follow
external packages dependencies since the installation procedure of that
external package should also install its dependencies (or make sure that
they are available).

Moreover for external packages, I hope that I'm not supposed to declare its
transitive dependencies as dependencies of my package, otherwise that
means that a change in an external package several transitive
dependencies down would force me to also update my package's
dependencies?

I don't think I've ever used lock directories so maybe there is a
special use-case here that is forcing this design?