diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md new file mode 100644 index 00000000000..7d293aab6b4 --- /dev/null +++ b/doc/dev/tools-design/implementation.md @@ -0,0 +1,1684 @@ +## Tools Implementation + +Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) + +## Summary + +This document specifies the implementatino of dune's tool management system. + + + + +**Table of Contents** + +- [Tools Implementation](#tools-implementation) +- [Summary](#summary) +- [How to Read This Document](#how-to-read-this-document) +- [Design](#design) + - [The `(tool)` stanza](#the-tool-stanza) + - [Syntax](#syntax) + - [Fields](#fields) + - [Compiler matching](#compiler-matching) + - [Examples](#examples) + - [Invalid tool selections](#invalid-tool-selections) + - [CLI commands](#cli-commands) + - [Version syntax](#version-syntax) + - [Batch operations](#batch-operations) + - [Open question: version separator](#open-question-version-separator) + - [Open question: `add` vs `install` naming](#open-question-add-vs-install-naming) + - [Open question: `dune tools run` PATH fallback](#open-question-dune-tools-run-path-fallback) + - [Tool resolution](#tool-resolution) + - [Source resolution](#source-resolution) + - [Version selection](#version-selection) + - [Example scenarios](#example-scenarios) + - [Directory structure](#directory-structure) + - [Tool pforms TODO](#tool-pforms-todo) + - [Syntax](#syntax-1) + - [Resolution](#resolution) + - [Examples](#examples-1) + - [Future: library access](#future-library-access) +- [Relationship to Package Management](#relationship-to-package-management) + - [Orthogonality: a key design principle](#orthogonality-a-key-design-principle) + - [The package-tool continuum](#the-package-tool-continuum) + - [Key differences from project dependencies](#key-differences-from-project-dependencies) + - [Why tools don't use `dune.lock/`](#why-tools-dont-use-dunelock) + - [Locking vs building](#locking-vs-building) + - [Future: Relocatable compiler and caching](#future-relocatable-compiler-and-caching) + - [Potential future stanza options](#potential-future-stanza-options) +- [GitHub Issues Addressed](#github-issues-addressed) + - [Arbitrary tool packages ([#12913])](#arbitrary-tool-packages-12913) + - [Lock directory clutter ([#10955], [#12097])](#lock-directory-clutter-10955-12097) + - [Compiler ABI mismatch ([#11229])](#compiler-abi-mismatch-11229) + - [Graceful fallback for missing tools ([#10578])](#graceful-fallback-for-missing-tools-10578) + - [Version constraints ([#12777])](#version-constraints-12777) + - [Version consistency ([#10688])](#version-consistency-10688) + - [Dev tools redesign ([#12914])](#dev-tools-redesign-12914) + - [Tool dependency isolation ([#12551])](#tool-dependency-isolation-12551) + - [No project build triggers ([#11037])](#no-project-build-triggers-11037) + - [Compiler-independent formatters ([#11038])](#compiler-independent-formatters-11038) + - [Version from config files ([#5315])](#version-from-config-files-5315) + - [Version constraint parsing ([#12866])](#version-constraint-parsing-12866) + - [Opam with-dev-setup integration ([#12135])](#opam-with-dev-setup-integration-12135) + - [Helpful error messages ([#13235], [#12975])](#helpful-error-messages-13235-12975) + - [Batch installation ([#12557])](#batch-installation-12557) + - [Failure isolation ([#12818])](#failure-isolation-12818) + - [Utop findlib integration ([#13471])](#utop-findlib-integration-13471) +- [Comparison with Other Tools](#comparison-with-other-tools) + - [uv (Python)](#uv-python) + - [cargo (Rust)](#cargo-rust) + - [cargo-run-bin (Rust community tool)](#cargo-run-bin-rust-community-tool) + - [npm (JavaScript)](#npm-javascript) + - [How Our Design Compares](#how-our-design-compares) + - [Tools compared](#tools-compared) + - [Comparison table](#comparison-table) + - [Project-local](#project-local) + - [Multi-version](#multi-version) + - [Declarative config](#declarative-config) + - [Compiler matching](#compiler-matching-1) + - [PATH fallback](#path-fallback) + - [Precompiled binaries](#precompiled-binaries) + - [Tool isolation](#tool-isolation) + - [Ephemeral runs](#ephemeral-runs) + - [Batch install](#batch-install) + - [Binary selection](#binary-selection) + - [Tool discovery](#tool-discovery) + - [Tool upgrade](#tool-upgrade) + - [System-wide install](#system-wide-install) + - [Recommendations from Research](#recommendations-from-research) +- [Learnings from the Prototype](#learnings-from-the-prototype) + - [What Worked Well](#what-worked-well) + - [Problems Discovered](#problems-discovered) + - [Reference Packages for Testing](#reference-packages-for-testing) +- [Ideas](#ideas) + - [Editor integration](#editor-integration) + + + +## How to Read This Document + +[Design](#design) details _how_ the system implements the requirements: +stanza syntax, CLI commands, version resolution algorithm, and directory +structure. Each section notes which requirements it implements. Open questions +are marked where decisions are pending. + +[Relationship to Package Management](#relationship-to-package-management) +explains the orthogonality principle and how tools differ from project +dependencies. + +[Comparison with Other Tools](#comparison-with-other-tools) analyzes how +`uv`,`cargo`, `cargo-run-bin`, and `npm` handle tool management, informing our +design decisions. + +## Design + +### The `(tool)` stanza + +_Implements: [1.3 Version specification](./requirements.md#14-version-specification), +[1.6 Binary selection](./requirements.md#16-binary-selection), +[3.1 Compiler compatibility](./requirements.md#31-compiler-compatibility), +[4.2 Persistent configuration](./requirements.md#42-persistent-configuration)_ + +Tools are declared in `dune-workspace` using the `(tool)` stanza. The stanza +uses the same dependency syntax as `(depends ...)` for the package field. + +#### Syntax + +```lisp +(tool + (package ) ; required: the tool package + (executables ...) ; optional: binaries to expose + (inherit_lock_dir []) ; optional: inherit solve env from a lock_dir stanza + (repositories |:inherit ...) ; optional: repositories to solve from + (solver_env ) ; optional: solver variable assignments + (unset_solver_vars |:inherit ...) ; optional: solver variables to leave undefined + (version_preference ) ; optional + (solve_for_platforms |:inherit ...) ; optional: platforms to solve for + (constraints |:inherit ...) ; optional: constraints on each solve + (pins |:inherit ...) ; optional: pins (names of (pin ...) stanzas) + (skip_compiler_match)) ; optional: disable compiler matching +``` + +CR-soon Sudha247: Can we express skip_compiler_match within the constraints field somehow? + +CR-soon Sudha247: do we need a `depopts` field? + +CR-soon Alizter: Edge cases for stanza parsing: + +- Duplicate stanzas for same package with different constraints: error, or + merge? +- Invalid package name syntax: error message? +- Stanza references non-existent repository: when to error (parse time vs solve + time)? + +CR-soon Sudha247: Capture constraints here? Related issue: +[dune#12777](https://github.com/ocaml/dune/issues/12777) + +#### Semantics of `:inherit` + + - Valid only in list-valued fields (repositories, unset_solver_vars, + solve_for_platforms, constraints, pins), and expands in place to the + corresponding field's value from the stanza named by `inherit_lock_dir`, e.g. + (repositories :inherit my-repo) means the inherited repositories followed by + my-repo. + - Omitting a field entirely means fully inherit it (when inherit_lock_dir is + present); writing the field without :inherit means fully override it. + - Using :inherit without inherit_lock_dir is an error. + - solver_env and version_preference are scalar-ish, so they follow the + omit-to-inherit / write-to-override rule with no :inherit keyword. + - Having an empty field with an existing (inherit_lock_dir) means omit that + particular field + +#### Fields + +- **`(package )`** (required): The opam package providing the tool. Accepts + either a plain package name or a name with version constraint: + - Plain: `ocamlformat` + - With constraint: `(ocamlformat (= 0.26.2))` + - Version operators: `=`, `<>`, `<`, `>`, `<=`, `>=` + - Conjunctions: `(and (>= 0.25.0) (< 0.27.0))` + +- **`(executables )`** (optional): The binary to run. Defaults to the + package name. If the package installs multiple binaries and no binary matches + the package name, omitting this field is an error. + +- **`(inherit_lock_dir [])`** (optional): Inherit the solve + environment from the `(lock_dir)` stanza whose path is `` + (defaulting to `dune.lock`, the default lock directory). The inheritable + fields are `repositories`, `solver_env`, `unset_solver_vars`, + `version_preference`, `solve_for_platforms`, `constraints`, and `pins`. + + When this field is present, each inheritable field follows one rule: + + | You write | Meaning | + |--------------------------|----------------------------------------------| + | *(field omitted)* | fully inherit the lock_dir's value | + | `(field )` | fully override; inheritance suppressed | + | `(field :inherit ...)` | splice the lock_dir's value in at that position | + | `(field :standard ...)` | splice dune's built-in default in | + + `:inherit` is only valid in list-valued fields, and only when + `inherit_lock_dir` is present; using it without is an error. Without + `inherit_lock_dir`, omitted fields take Dune's ordinary defaults. + +- **`(repositories |:inherit|:standard ...)`** (optional): + Repositories to solve from, as an ordered set. `:standard` is the default + repository set; `:inherit` is the lock_dir's repositories. Defaults to + `:inherit` when `inherit_lock_dir` is present, `:standard` otherwise. + +- **`(solver_env )`** (optional): Solver variable assignments for + this tool's solve. Scalar semantics: omitted = inherited (when + `inherit_lock_dir` is present), written = taken exactly as written, + `(solver_env)` = empty. To inherit the environment *minus* a variable, + inherit it and list the variable in `unset_solver_vars`. + +- **`(version_preference )`** (optional): Whether the solver + prefers the newest or oldest satisfying versions. Omitted = inherited + (when `inherit_lock_dir` is present), otherwise `newest`. To suppress + inheritance, spell the default: `(version_preference newest)`. + +- **`(solve_for_platforms |:inherit|:standard ...)`** (optional): + Platforms to solve this tool for. `:standard` is the default platform set; + `:inherit` is the lock_dir's platforms. Writing the field so that it + evaluates to an empty set is an error. To get the default platforms + explicitly, write `(solve_for_platforms :standard)`. + +- **`(constraints |:inherit ...)`** (optional): Additional constraints + on the solve, in `(depends ...)` syntax. A constraint only binds when the + named package actually appears in the tool's solution, so inheriting the + project's constraints is safe: irrelevant ones are no-ops, and shared + transitive dependencies stay consistent with the project. + +- **`(pins |:inherit ...)`** (optional): Names of `(pin ...)` + stanzas to apply to this tool's solve. + +- **`(skip_compiler_match)`** (optional): When present, disables the default + compiler matching behavior. Use this for tools that don't need compiler + compatibility (e.g., formatters that only parse source text). + +CR-soon Alizter: The `(repositories)` field is parsed but verify that the +restriction is actually applied during solving in the prototype. + +CR-soon Alizter: The prototype uses `compiler_compatible` (opt-in) but this doc +says `skip_compiler_match` (opt-out). Reconcile naming and semantics. + + +#### Compiler matching + +_Implements: [3.1 Compiler compatibility](./requirements.md#31-compiler-compatibility)_ + +Tools that read project build artifacts (e.g., ocamllsp reading `.cmi` files) +must be built with a compatible compiler to avoid ABI mismatches. By default, +tools are built with a compiler matching the environment. + +The compiler is detected in this order: + +1. **Project compiler** - if using dune package management with a `dune.lock/` +2. **System OCaml** - from opam switch or PATH +3. **Any compatible version** - if no compiler is detected + +When `(skip_compiler_match)` is set, the tool is solved without compiler +constraints, allowing it to use any OCaml version. + +CR-soon Alizter: Edge cases for compiler matching: + +- No compiler found (no opam switch, no system OCaml): error, or pick any? +- System OCaml version changes between `add` and `run`: rebuild, or warn? +- Tool requires newer OCaml than project: error message? + +#### Examples + +```lisp +;; Minimal: just the package name +(tool + (package ocamlformat)) + +;; With version constraint +(tool + (package + (ocamlformat + (= 0.26.2)))) + +;; With version range +(tool + (package + (ocamlformat + (and + (>= 0.25.0) + (< 0.27.0))))) + +;; Specifying the binary for multi-binary packages +(tool + (package menhir) + (executables menhirSdk)) + +;; Tool that doesn't need compiler matching +(tool + (package ocamlformat) + (skip_compiler_match)) + +;; Tool pinned to a fork +(pin + (name ocamlformat) + (url "git+https://github.com/me/ocamlformat#my-fork")) + +(tool + (package ocamlformat) + (pins ocamlformat)) + +;; Inherit from lock_dir +(tool + (package ocamlformat) + (inherit_lock_dir)) + +;; More involved menhir + +(tool + (package (menhir (>= 20220210))) + (executables menhir menhirSdk) + (inherit_lock_dir dev.lock) + (repositories :inherit overlay-repo) ; dev.lock's repos, then overlay-repo + (constraints :inherit (dune-build-info (>= 3.0))) ; project constraints plus one + (solver_env + (with-test false)) ; full override: dev.lock's solver_env + ignored + (version_preference oldest) + (solve_for_platforms :standard) ; opt out of dev.lock's platforms, use dune's + defaults + (skip_compiler_match)) + +``` + +#### Invalid tool selections + +Some packages cannot be used as tools: + +- **Library-only packages**: Packages that provide no executables (e.g., `base`, + `lwt`) cannot be tools. Error at `dune tools add` time. + +CR-someday Alizter: The `%{tool:lib:...}` pform in [Tool pforms](#tool-pforms) +could enable library-only tool packages, but this needs further design work. + +CR-soon Alizter: How can we error at dune tools add time? We would only find out +after building the tool. + +- **Compiler packages**: Selecting `ocaml-base-compiler` or `ocaml-system` as a + tool is invalid. These conflict with the compiler matching logic. + +- **Dune itself**: Using `dune` as a tool is the "in and out" problem + ([non-requirement 6.3](#63-tools-for-tools)). Circular dependency issues make + this out of scope. + +CR-soon Alizter: The explanations for compiler packages and dune itself need +more thought. + +CR-soon Alizter: Specify exact error messages for these cases. + +### CLI commands + +_Implements: [1.4 Version specification](./requirements.md##14-version-specification), +[1.6 Binary selection](./requirements.md#16-binary-selection), +[2.3 Programmatic use](./requirements.md#23-programmatic-use), [4.1 CLI](./requirements.md#41-cli)_ + +CR-soon Alizter: The CLI commands are generally underspecified. Each command +needs detailed behavior documentation: what triggers downloads/builds, how +errors are reported, interaction with stanzas vs CLI-added tools, exit codes, +output format, etc. + +``` +dune tools List all locked tools (alias for list) +dune tools list List all locked tools and versions + +dune tools add Lock all tools from (tool) stanzas +dune tools add [.] ... Lock specific tool(s) + +dune tools run [.] [--bin ] [-- ] + Run a tool, passing arguments after -- + +dune tools path Print path to the managed binary directory (or directories) +dune tools path [.] [--bin ] Print path to tool executable + +dune tools remove [.] ... Remove a tool's lock directory +``` + +The `--bin` flag is required when a package provides multiple binaries and no +`(executable ...)` is specified in the `(tool)` stanza. + +CR-soon Alizter: Edge cases for `dune tools add`: + +- Package doesn't exist in repository: error message? +- Network unavailable: error message, offline mode? +- Solver fails (unsatisfiable constraints): error with diagnostics? +- Build fails: partial state cleanup? +- Interrupted mid-install: cleanup, or resume on next run? +- Already locked at same version: no-op, or re-solve? +- Already locked at different version: add second version, or replace? + +CR-soon Alizter: The prototype has `--allow-not-installed` flag for +`dune tools path`. Document this flag and its use case (editor integration). + +CR-soon Alizter: `dune tools path` behavior is underspecified: + +- Should `path` trigger download and build if not yet built? Or only return path + if already built, erroring otherwise? +- If `--bin` is specified, can we compute the path without building (since path + is deterministic from package/version/binary name)? +- If package has multiple binaries and no `--bin`: error immediately, or + download and build first to discover available binaries? +- Tool not locked: error, or fall back to `which`? +- How does `path` interact with `(tool)` stanzas vs CLI-added tools? Same + resolution as `run`? + +CR-soon Sudha247: We now have two sources for the current state of what tools +exist - one in the declarative config within dune-workspace, and one in the +state of tools installed and their lockdirs. Ideally we should just have a +single source of truth for what tools should be installed. We can do this by +making the CLI edit dune-workspace rather than operating on its own. A +pre-requisite for this is: https://github.com/ocaml/dune/pull/13758. + +CR-someday Alizter: Consider a single bin directory with symlinks to all tool +executables (like npm's `node_modules/.bin/`). Instead of per-tool paths, have +`_build/.tools/bin/` containing symlinks to all installed tool binaries. This +simplifies editor integration (one directory to add to PATH), avoids PATH length +limits (especially on Windows), and provides a single stable location for +discovery. Trade-off: need to maintain symlinks as tools are added/removed. + +#### Version syntax + +The CLI uses dot-separated version syntax: `.` + +```bash +dune tools add ocamlformat # Latest version +dune tools add ocamlformat.0.26.2 # Specific version +``` + +#### Batch operations + +Batch operations apply to commands that manage tool state, not to commands that +use tools: + +| Command | Batch | Rationale | +| ------------------- | ----- | ----------------------------------------------- | +| `dune tools add` | ✓ | No args: all from stanzas; with args: specified | +| `dune tools remove` | ✓ | Multiple packages | +| `dune tools run` | ✗ | Running is inherently single-target | +| `dune tools path` | ✗ | Path lookup is per-tool | + +CR-soon Alizter: Edge cases for `dune tools remove`: + +- Tool not locked: error, or no-op with warning? +- Tool declared in stanza: remove lock but warn it will be re-added? +- Multiple versions locked: remove all, or require version specifier? +- Remove while tool is running: behavior? + +**`dune tools add`** has two modes: + +```bash +dune tools add # Lock all from (tool) stanzas +dune tools add ocamlformat odoc # Lock specific packages +dune tools add ocamlformat.0.26.2 odoc.2.4.0 # With version constraints +``` + +The no-argument form is critical for CI and onboarding - a single command to set +up all project tooling from declarative configuration. + +CR-someday Alizter: Integration with opam's `:with-dev-setup` marker ([#12135]): + +[#12135]: https://github.com/ocaml/dune/issues/12135 + +Projects often declare their development tools in opam files using the +`:with-dev-setup` filter (e.g., `ocamlformat :with-dev-setup`). Currently, dune +tools doesn't read these - users must manually add each tool or duplicate the +declarations in `(tool)` stanzas. + +One possible approach: + +```bash +dune tools add --with-dev-setup # Install tools from opam :with-dev-setup deps +dune tools add --with-doc # Install tools from :with-doc deps +``` + +This would enable: + +- Single source of truth for project tooling (opam file) +- Automatic tool discovery for new contributors +- CI scripts that don't need to know which tools to install + +CR-soon Alizter: Is `--with-dev-setup` the right approach? Alternatives: + +- Automatic: `dune tools add` (no args) always includes `:with-dev-setup` deps +- Separate command: `dune tools setup` specifically for opam marker integration +- No integration: keep `(tool)` stanzas as the only source, avoid duplication by + not using `:with-dev-setup` in opam files at all The flag approach adds + complexity. Need to evaluate if the use case justifies it. + +CR-soon Alizter: If we do integrate, design questions: + +- Read from `dune-project` `(depends)` or from `.opam` files? +- What if opam file says `ocamlformat` but `.ocamlformat` says `version=0.26.2`? +- How does this interact with `(tool)` stanzas that specify the same package? + +CR-soon Alizter: The prototype requires at least one package argument. The +no-argument batch mode (lock all from stanzas) is not yet implemented. + +#### Open question: version separator + +The current prototype uses `.` as the version separator (`pkg.version`), aligned +with opam conventions. However, `@` is common in other ecosystems: + +- **npm/npx**: `npx cowsay@1.5.0` +- **cargo**: `cargo install ripgrep@14.0.0` + +Opam package names cannot contain dots (allowed: `[a-zA-Z0-9_+-]+`), so `.` is +unambiguous. The choice is primarily about familiarity vs ecosystem consistency. + +#### Open question: `add` vs `install` naming + +The current prototype uses `dune tools add` for adding individual tools. +However, the naming convention is still open for discussion: + +- **Option A**: `add` for individual, `install` for batch (current prototype) +- **Option B**: `install` for both (consistent with legacy, npm, cargo) +- **Option C**: `lock` to emphasize the solving/locking aspect +- **Option D**: Semantic continuum where commands differ by how much work they + do: + - `add` = lock only (solve + write lock) + - `install` = lock + build (download + compile) + - `run` = lock + build + execute This gives users control over when work + happens (useful for CI, Docker builds). + +Considerations: + +- Legacy issues reference `dune tools install` +- `add` suggests incremental, `install` suggests setup +- `lock` aligns with `dune pkg lock` but may confuse (tools also build) +- Option D aligns with user expectations: "add to config" vs "install software" + +#### Open question: `dune tools run` PATH fallback + +Should `dune tools run ` fall back to system PATH when the tool is not +locked? + +- **Option A**: Fall back to PATH (consistent with build rules like `dune fmt`) +- **Option B**: Error with suggestion to lock or install on PATH (current impl) + +Considerations: + +- Falling back makes `dune tools run` a universal "run this tool" command +- Erroring is more explicit and avoids silently using unexpected versions +- Build rules need PATH fallback for gradual adoption (e.g., `dune fmt` without + locking ocamlformat) +- CLI commands could be stricter since the user explicitly requested a tool + +### Tool resolution + +_Implements: [1.5 Multi-version support](./requirements.md#15-multi-version-support), +[2.5 System PATH fallback](./requirements.md#25-system-path-fallback), +[5.1 Format rules](#51-format-rules)_ + +Tool resolution answers two questions: + +1. **Source**: Where does the tool come from? +2. **Version**: Which version of the tool? + +#### Source resolution + +When dune needs a tool, it checks these sources in order: + +1. **Workspace `(tool)` stanza** - declared in `dune-workspace` +2. **Locked versions** - previously added via `dune tools add` +3. **System PATH** - fallback if nothing else found + +The first source that has the tool wins. If no source has it, dune falls back to +system PATH (which may result in "command not found"). + +#### Version selection + +CR-soon Alizter: Version conflict resolution algorithm is undefined. What +happens when multiple sources specify conflicting versions? E.g., stanza says +`(= 0.26.2)` but `.ocamlformat` says `version=0.27.0`. Need to define precedence +or error behavior. + +How the version is determined depends on context: + +**CLI commands** (`dune tools run`, `dune tools add`): + +- Explicit version: `dune tools run ocamlformat.0.26.2` uses that version +- No version: uses the single locked version, or errors if multiple exist + +**`(tool)` stanza**: + +- Version constraint in stanza: `(package (ocamlformat (= 0.26.2)))` +- Constraint applied when locking, not at runtime + +**Formatting rules** (`dune fmt`): + +1. Read `version` field from `.ocamlformat` file +2. If version present: use `%{tool:ocamlformat.}` pform +3. If no version: fall back to `%{bin:ocamlformat}` (system PATH) + +The pform handles locking/building/PATH fallback per the resolution rules above. + +#### Example scenarios + +| Scenario | Source | Version | +| ------------------------------------------- | -------------- | -------------- | +| `(tool (package ocamlformat))` in workspace | stanza | locked version | +| `dune tools add ocamlformat.0.26.2` | locked | 0.26.2 | +| `.ocamlformat` says `version=0.26.2` | locked or PATH | 0.26.2 | +| No config, ocamlformat on PATH | PATH | system version | + +CR-soon Alizter: Specify error messages for failure modes: solver fails, network +error, build fails, binary not found, multiple binaries without --bin, etc. Is +the locked version the most recent? + +### Directory structure + +_Implements: [1.2 Workspace-local](./requirements.md#12-workspace-local), +[1.5 Clean source tree](./requirements.md#15-clean-source-tree), +[3.2 Dependency isolation](./requirements.md#32-dependency-isolation)_ + +Tools use three types of directories: + +1. **External lock directories** - Created by `dune tools add`, contain solver + output +2. **Build lock directories** - Internal copies for build rules +3. **Install directories** - Built artifacts (binaries, libraries) + +``` +_build/ +├── .tools.lock/ # External lock directories (solver output) +│ └── / +│ └── / +│ ├── lock.dune # Lock file +│ └── .pkg # Package metadata +└── _private/ + └── default/ + ├── .tool-locks/ # Build lock directories (internal) + │ └── / + │ └── / + │ ├── lock.dune + │ └── .pkg + └── .tools/ # Install directories (built artifacts) + └── / + └── / + └── target/ + ├── bin/ # Installed executables + └── cookie # Installation trace (artifacts + variables) +``` + +**Path flow**: External lock → copied to build lock → built to install dir. + +**Key paths**: + +- External lock: `_build/.tools.lock///` +- Build lock: `_build/_private/default/.tool-locks///` +- Install: `_build/_private/default/.tools///target/` + +Note: The external lock directory (`_build/.tools.lock/`) is at the build root, +not inside `_private/`. This allows tools to be locked without going through the +build system. The build lock directory and install directory are both inside +`_private/default/` as they are managed by the build system. + +### Tool pforms TODO + +CR-Shon: I don't understand why this is necessary: why can't actions just keep +using the `bin` pform? Introduction of yet another configuration quirk should +have very good motive, but we already have a way to refer to executables, and +tools are just a way to manage the installation of executables. + +Maybe what we actually want is just way of specifying the scope of tool +versions? Perhaps via an extension to `env`: this seems like an `env` issue... + +_Implements: [5.4 Tool references in actions](#54-tool-references-in-actions)_ + +Build actions can reference tool executables via `%{tool:...}` pforms. + +#### Syntax + +``` +%{tool:} ; executable (default binary) +%{tool:.} ; specific version +``` + +#### Resolution + +1. If tool is locked, use the locked executable path +2. If tool is not locked, fall back to PATH (like `%{bin:...}`) + +CR-soon Alizter: What happens when version is omitted and multiple versions are +locked? Error, or pick one? + +#### Examples + +```lisp +;; Run a tool executable +(rule + (action (run %{tool:ocamlformat} %{deps}))) + +;; Use a specific version +(rule + (action (run %{tool:ocamlformat.0.26.2} %{deps}))) +``` + +#### Future: library access + +A future extension could allow accessing libraries from tool packages via +`%{tool:lib:}`. This would enable using a library solved independently from +the project's dependency graph. + +However, this has significant drawbacks: if two tool libraries share a common +dependency (e.g., both depend on `base`), they are solved independently and may +have incompatible versions. Using both in the same build rule could cause +linking errors. This needs further design work around "tool groups" or coherence +constraints before being viable. + +## Relationship to Package Management + +Tools are built on the same infrastructure as `dune pkg` (Dune's package +management): + +- Same solver (`opam-0install-solver`) +- Same repository system (opam-repository, overlays, custom repos) +- Same lock file format +- Same build rules for fetching and compiling packages + +### Orthogonality: a key design principle + +Tools are **orthogonal** to project package management. This decoupling +simplifies the design: + +- **No project lock required**: Tools work even without `dune.lock/`. A project + can use `dune tools` without enabling package management. +- **Independent solving**: Adding/removing tools doesn't affect project + dependencies, and vice versa. +- **No build invalidation**: Tool changes don't trigger project rebuilds + ([requirement 3.3](#33-no-build-triggers)). +- **Multi-version is natural**: Since each tool version is solved independently, + supporting multiple versions falls out of the design. + +The legacy system violated this principle by mixing tool dependencies with +project packages (the "wrapper package hack"), causing solver conflicts and +complex regeneration logic. + +### The package-tool continuum + +Rather than a binary choice between "package" and "tool", the design supports a +continuum of integration levels: + +**Fully integrated (packages)**: Dependencies in `dune.lock/` are solved +together, libraries and executables are available throughout the build, and +changes affect the entire project. + +**Fully isolated (tools)**: Each tool has its own lock directory, is solved +independently, and by default contributes nothing to the build environment. The +tool sits in `_build/` invisible to the project unless explicitly invoked. + +**Selective exposure (via pforms)**: The `%{tool:...}` pform +([Tool pforms](#tool-pforms)) allows build rules to selectively reference tool +executables. This lets a tool remain isolated (independently solved, no +dependency conflicts) while still being usable in specific rules. + +This continuum lets users choose the right level of integration: use packages +when you need tight coupling and shared dependencies; use tools when you want +isolation and version flexibility; use pforms when you need something in +between. + +### Key differences from project dependencies + +| Aspect | Project packages (`dune pkg`) | Tools (`dune tools`) | +| ------------- | --------------------------------- | ---------------------------------- | +| Lock location | `dune.lock/` (committed) | `_build/.tools.lock/` (ephemeral) | +| Solving | All packages solved together | Each tool solved independently | +| User workflow | `dune pkg lock` then `dune build` | `dune tools add`, builds on demand | +| Compiler | Project's compiler | Matches project or system OCaml | +| Requires lock | Yes | No (can use system OCaml) | + +CR-soon Alizter: system-ocaml, the opam package, is a bit of a hack. If a user +changes their opam switch it might not work as intended. + +CR-soon what does the Requires lock even mean here? + +### Why tools don't use `dune.lock/` + +Tools are stored in `_build/` rather than alongside `dune.lock/` because: + +1. **Tools are workspace infrastructure**, not project deliverables +2. **Different sub-projects may need different tool versions** (e.g., different + `.ocamlformat` versions per directory) +3. **Avoids cluttering source tree** (addresses [#10955], [#12097]) + +[#10955]: https://github.com/ocaml/dune/issues/10955 +[#12097]: https://github.com/ocaml/dune/issues/12097 + +The tradeoff is that tool lock directories are not committed to version control. +Reproducibility comes from `(tool)` stanzas in `dune-workspace`, which declare +version constraints that are re-solved when `_build/` is deleted. + +### Locking vs building + +These are separate phases, though `dune tools add` combines them: + +1. **Locking**: Solve dependencies, write lock directory with package metadata +2. **Building**: Download sources, compile tool, install to target directory + +Building (which includes downloading sources) happens lazily on first +`dune tools run`, not during `dune tools add`. The `add` command only solves +dependencies and writes the lock directory. + +**Open question**: How can users download, build, and install tools without +running them? This is useful for CI (prepare all tools upfront) or offline +scenarios. Options include a `--build` flag on `add`, a separate +`dune tools build` command, or relying on `dune tools path` to trigger builds. + +### Future: Relocatable compiler and caching + +Currently, each workspace builds its own copy of tools, even if the same tool +version is used across multiple projects. This is wasteful. + +The **relocatable compiler** work will enable: + +- **Shared build cache**: Tool builds can be cached and reused across projects +- **Faster tool installation**: If another project already built + `ocamlformat.0.26.2` with the same compiler, reuse the cached artifacts +- **Binary distribution**: Pre-built tool binaries could be distributed via + binary package repositories + +This will address the performance concerns of +[non-requirement 6.1](#61-system-wide-installation) (System-wide installation) +while maintaining project isolation. + +### Potential future stanza options + +The `(lock_dir)` stanza for project packages supports options not yet available +for tools: + +- `(solver_env ...)` - inject variables into solver +- `(constraints ...)` - additional solver constraints +- `(version_preference newest|oldest)` - prefer newest or oldest versions +- `(pins ...)` - pin packages to specific sources + +These may be added to `(tool)` stanzas if use cases emerge. In particular, +`(pins ...)` would help with [non-requirement 6.3](#63-tools-for-tools) (Tools +for tools) by allowing tools to be pinned to local paths. + +## GitHub Issues Addressed + +This section maps known issues to the requirements that address them. + +### Arbitrary tool packages ([#12913]) + +**Problem**: The current design requires every tool to be hardcoded. Only a few +tools are supported, and adding new ones requires code changes to dune. + +**Addressed by**: Requirement 1.1 (Generality). Any opam package that provides +an executable can be installed as a tool via `dune tools add `. + +[#12913]: https://github.com/ocaml/dune/issues/12913 + +### Lock directory clutter ([#10955], [#12097]) + +**Problem**: Users complained about lock directories cluttering the source tree. + +**Addressed by**: Requirement 1.5 (Clean source tree). Tool lock directories are +now stored in `_build/.tools.lock/`, not the source tree. + +[#10955]: https://github.com/ocaml/dune/issues/10955 +[#12097]: https://github.com/ocaml/dune/issues/12097 + +### Compiler ABI mismatch ([#11229]) + +**Problem**: Pre-built tool binaries (musl toolchain) don't work with +locally-built OCaml compilers because interface hashes don't match. + +**Addressed by**: Requirement 3.1 (Compiler compatibility). Tools are built from +source with matching compiler constraints by default. + +[#11229]: https://github.com/ocaml/dune/issues/11229 + +### Graceful fallback for missing tools ([#10578]) + +**Problem**: `dune fmt` fails hard when ocamlformat isn't installed, even if +other formatters could run. + +**Addressed by**: Requirements 2.4 (System PATH fallback) and 5.1 (Format +rules). When no version is specified in `.ocamlformat`, dune falls back to PATH. +If not found, the OCaml formatting rules are skipped rather than failing the +entire build. + +[#10578]: https://github.com/ocaml/dune/issues/10578 + +### Version constraints ([#12777]) + +**Problem**: No robust way to specify version constraints for dev tools. Users +had to hardcode paths to lock directories. + +**Addressed by**: Requirements 1.3 (Version pinning) and 4.2 (Persistent +configuration). The `(tool)` stanza allows declarative version constraints: + +```lisp +(tool + (package + (ocamlformat + (= 0.26.2)))) +``` + +[#12777]: https://github.com/ocaml/dune/issues/12777 + +### Version consistency ([#10688]) + +**Problem**: When ocamlformat isn't a project dependency, `dune fmt` captures +whatever version is on PATH, leading to inconsistent formatting. + +**Addressed by**: Requirements 1.3 (Version pinning) and 5.1 (Format rules). +When `.ocamlformat` specifies a version, dune uses the locked version. The PATH +is only used as fallback when no version is specified. + +[#10688]: https://github.com/ocaml/dune/issues/10688 + +### Dev tools redesign ([#12914]) + +**Problem**: Meta-issue tracking the need for a complete redesign of dev tools. + +**Addressed by**: This entire design document. Key improvements: + +- Generality (1.1): Any package, not just hardcoded ones +- Clean source tree (1.5): Lock dirs in `_build/` +- Version pinning (1.3): Declarative constraints via stanzas +- Multi-version (1.4): Different projects can use different versions +- Compiler compatibility (3.1): Tools built with matching compiler + +[#12914]: https://github.com/ocaml/dune/issues/12914 + +### Tool dependency isolation ([#12551]) + +**Problem**: Tools and project packages conflict when they share dependencies. + +**Addressed by**: Requirements 3.2 (Dependency isolation) and 5.3 (REPL). Each +tool has its own lock directory and is solved independently from project +dependencies. + +[#12551]: https://github.com/ocaml/dune/issues/12551 + +### No project build triggers ([#11037]) + +**Problem**: Running `dune fmt` triggers a full build of project dependencies, +even though formatting only needs ocamlformat. + +**Addressed by**: Requirements 3.3 (No build triggers) and 5.1 (Format rules). +Tool operations are isolated from project builds. + +[#11037]: https://github.com/ocaml/dune/issues/11037 + +### Compiler-independent formatters ([#11038]) + +**Problem**: `dune fmt` fails if no OCaml compiler is available, even though +ocamlformat could be a standalone binary. + +**Addressed by**: Requirement 3.1 (Compiler compatibility). Tools match the +project or system compiler, but formatters can opt out via +`(skip_compiler_match)`. + +[#11038]: https://github.com/ocaml/dune/issues/11038 + +### Version from config files ([#5315]) + +**Problem**: The `version` field in `.ocamlformat` should determine which +ocamlformat version to use. Different projects within a workspace may have +different `.ocamlformat` files specifying different versions. + +**Addressed by**: Requirements 1.3 (Version pinning), 1.4 (Multi-version +support), and 5.1 (Format rules). Formatting rules resolve the correct +ocamlformat version based on each directory's `.ocamlformat` file, and multiple +versions can coexist in the workspace. + +[#5315]: https://github.com/ocaml/dune/issues/5315 + +### Version constraint parsing ([#12866]) + +**Problem**: Version constraints for dev tools don't work correctly. + +**Addressed by**: Requirement 1.3 (Version pinning). The new `(tool)` stanza +properly supports version constraints using dependency syntax. + +[#12866]: https://github.com/ocaml/dune/issues/12866 + +### Opam with-dev-setup integration ([#12135]) + +**Problem**: Projects declare development tools in opam files using the +`:with-dev-setup` filter, but dune tools doesn't integrate with this. Users must +manually discover and add each tool, or duplicate declarations in both opam +files and `(tool)` stanzas. + +**Addressed by**: Requirement 4.1 (CLI), future work. See "Batch operations" in +the Specification for the planned `--with-dev-setup` flag. + +[#12135]: https://github.com/ocaml/dune/issues/12135 + +### Helpful error messages ([#13235], [#12975]) + +**Problem**: When tools aren't installed, error messages don't guide users +toward the correct commands. + +**Addressed by**: Requirement 4.1 (CLI). Commands should provide helpful +suggestions when tools aren't installed. + +[#13235]: https://github.com/ocaml/dune/issues/13235 +[#12975]: https://github.com/ocaml/dune/issues/12975 + +### Batch installation ([#12557]) + +**Problem**: Installing multiple tools requires multiple commands. + +**Addressed by**: Requirement 4.1 (CLI). `dune tools add` accepts multiple +package arguments, and with no arguments locks all tools from `(tool)` stanzas. +See "Batch operations" in the Specification. + +[#12557]: https://github.com/ocaml/dune/issues/12557 + +### Failure isolation ([#12818]) + +**Problem**: Failed tool installation can leave the environment in a broken +state. + +**Addressed by**: Requirement 3.2 (Dependency isolation). Each tool is isolated, +so a failed install doesn't affect other tools or the project. + +[#12818]: https://github.com/ocaml/dune/issues/12818 + +### Utop findlib integration ([#13471]) + +**Problem**: Running `#require` in utop triggers findlib warnings about +non-existent directories from dune's sandboxed package paths. + +**Addressed by**: Requirement 5.3 (REPL). When utop is configured as a tool, it +should be properly integrated with dune's package environment. + +[#13471]: https://github.com/ocaml/dune/issues/13471 + +## Comparison with Other Tools + +This section documents how other ecosystems handle tool management, informing +our design decisions and identifying gaps in our approach. + +### uv (Python) + +[uv][uv docs] distinguishes between ephemeral and persistent tool usage: + +- **`uvx `**: Runs tool in temporary virtual environment, cached but + disposable +- **`uv tool install `**: Persistent installation, executables on PATH + +**Key features**: + +- Per-tool isolated virtual environments (no cross-tool conflicts) +- Version syntax: `uvx ruff==0.3.0` (pinned), `uvx ruff@latest` (latest only) +- Constraint preservation: `uv tool install black>=23,<24` is respected by + `uv tool upgrade` + +**Relevance to requirements**: + +- 2.1: uv support adding its binary directory to shells https://docs.astral.sh/uv/concepts/tools/#tool-executables +- 2.2: + - uv uses a simple, predictable location for binary installations + https://docs.astral.sh/uv/reference/storage/#tool-executables and it allows + - uv allows configuration of the installation path and an API for locating the + bin directory https://docs.astral.sh/uv/reference/storage/#tool-executables +- **3.2 Dependency isolation**: Per-tool venvs achieve isolation. Our separate + lock dirs provide similar isolation. + +**Gap**: No project-local tool declaration. Tools are always user-global. This +violates our [requirement 1.2](#12-workspace-local) (Workspace-local). + +**What we should adopt**: + +- `upgrade` command that respects original constraints +- Ephemeral vs persistent distinction + +CR-soon Alizter: `dune tools upgrade` is not in the CLI specification. Add it or +note as future work. + +### cargo (Rust) + +[cargo install][cargo docs] provides minimal built-in tool support: + +- All binaries installed to single `~/.cargo/bin/` directory +- No isolation between tools +- No multi-version support (new version overwrites old) +- No project-local tools + +**Key features**: + +- Version constraint syntax: `cargo install ripgrep@1.2.0`, `--version ~1.2` +- Smart reinstall: only rebuilds if version/features/profile changed +- `cargo install --list` shows all installed packages + +**Relevance to requirements**: + +- **1.2 Workspace-local**: Violated. Cargo's model is entirely system-wide. +- **3.2 Dependency isolation**: Violated. All tools share one bin directory. + +**Gap**: No project-local tools, which is why cargo-run-bin exists. + +### cargo-run-bin (Rust community tool) + +[cargo-run-bin] is the **closest analog to our design**. It fills cargo's gaps +with project-local tool management: + +```toml +[package.metadata.bin] +cargo-nextest = { version = "0.9.57", locked = true } +dprint = { version = "0.30.3" } +cargo-mobile2 = { version = "0.5.2", bins = ["cargo-android", "cargo-mobile"] } +``` + +**Key features**: + +- Project-local `.bin/` cache directory +- Declarative config in `Cargo.toml` metadata section +- `bins` array for multi-binary packages +- `locked` flag for reproducible dependency resolution +- `cargo bin --install` to install all configured tools at once + +**Relevance to requirements**: + +- **1.2 Workspace-local**: ✓ Achieves this via `.bin/` directory +- **4.2 Persistent configuration**: ✓ Uses `Cargo.toml` metadata + +**What we should adopt**: + +- `bins` field for explicit binary listing (vs auto-discovery) +- `--locked` flag for reproducibility +- **Batch install command** (`dune tools add`). Critical for CI + +### npm (JavaScript) + +[npm][npm docs] has the most mature approach for project-local tools: + +- Tools declared in `devDependencies` in `package.json` +- Binaries linked to `node_modules/.bin/` +- `npx` runs binaries, preferring local over global +- npm scripts automatically get `node_modules/.bin` in PATH +- `package-lock.json` provides exact version pinning + +**Relevance to requirements**: + +- **1.2 Workspace-local**: ✓ `node_modules/.bin/` is project-local +- **2.2 Programmatic use**: ✓ Single directory for all tool binaries +- **4.2 Persistent configuration**: ✓ `devDependencies` in package.json + +**What we should adopt**: + +- devDependencies pattern (tools as dev deps, committed to repo) +- Auto-PATH for scripts (tools available without explicit path) +- `npx` prefers local, falls back to fetch + +### How Our Design Compares + +**Note**: Dune entries below describe **design requirements**, not necessarily +fully implemented features. Each entry references the relevant requirement +(e.g., "Req 1.2") to enable verification against the specification. + +#### Tools compared + +**uv** (Python): Fast Python package manager with tool support. Tools are +installed globally with per-tool isolated virtual environments. Supports +ephemeral runs (`uvx`) and constraint-preserving upgrades. No project-local or +declarative configuration. ([uv docs]) + +**cargo** (Rust): Rust's built-in package manager. Minimal tool support. All +binaries go to `~/.cargo/bin/`, no isolation, no multi-version. Smart reinstall +detects version changes. Compiler matching via rustup's `rust-toolchain.toml`. +([cargo docs]) + +**cargo-run-bin** (Rust): Community tool filling cargo's gaps. Project-local +`.bin/` cache, declarative config in `Cargo.toml` metadata, batch install. +Closest analog to our design in the Rust ecosystem. ([cargo-run-bin]) + +**npm** (JavaScript): Most mature project-local approach. Tools in +`devDependencies`, binaries in `node_modules/.bin/`, `npx` for execution. +Semver-respecting upgrades, lock file for reproducibility. ([npm docs]) + +#### Comparison table + +| Feature | dune | uv | cargo | cargo-run-bin | npm | +| -------------------- | ---- | --- | ----- | ------------- | --- | +| Project-local | ✓ | ✗ | ✗ | ✓ | ✓ | +| Multi-version | ✓ | ✗ | ✗ | ✗ | ✓\* | +| Declarative config | ✓ | ✗ | ✗ | ✓ | ✓ | +| Compiler matching | ✓ | ✗ | ✓\* | ✗ | ✗ | +| PATH fallback | ✓ | ✗ | ✗ | ✗ | ✓ | +| Precompiled binaries | ✗ | ✓ | ✗ | ✓\* | ✓ | +| Tool isolation | ✓ | ✓ | ✗ | ✓ | ✓ | +| Ephemeral runs | ✗ | ✓ | ✗ | ✗ | ✓ | +| Batch install | ✓ | ✗ | ✗ | ✓ | ✓ | +| Binary selection | ✓ | ✓ | ✓ | ✓ | ✓ | +| Tool discovery | ✓ | ✓ | ✗ | ✗ | ✓ | +| Tool upgrade | ✗ | ✓ | ✓\* | ✗ | ✓ | +| System-wide install | ✗ | ✓ | ✓ | ✗ | ✓ | + +_Asterisks indicate caveats; see relevant subsection for details._ + +#### Project-local + +Tools are installed within the project directory, isolated from other projects +and the global system. + +**Dune** (Req 1.2, 1.5): Tools are workspace-local. Lock files stored in +`_build/.tools.lock///`, built artifacts in +`_build/_private/default/.tools///`. + +**uv**: ✗ Tools installed globally. "Each tool environment is linked to a +specific Python version." Multiple Python versions require separate installs +with `--python` flag. If the Python version is uninstalled, the tool breaks. +([uv docs]) + +**cargo**: ✗ All binaries installed to single `~/.cargo/bin/`, shared by all +projects. Only one version per tool; reinstalling overwrites. Different versions +require custom `CARGO_INSTALL_ROOT` per project. ([cargo docs]) + +**cargo-run-bin**: ✓ Project-local `.bin/` cache directory. "Designed to manage +your Rust binaries within the context of your project." ([cargo-run-bin]) + +**npm**: ✓ Binaries linked to `node_modules/.bin/` per project. "Executables +linked into `./node_modules/.bin`." ([npm docs]) + +#### Multi-version + +Multiple versions of the same tool can coexist and be used within a single +workspace. + +**Dune** (Req 1.4): Versioned paths `_build/.tools.lock///` allow +multiple versions. Version selection: if 1 version locked, use it; if multiple, +error unless specified (e.g., `.ocamlformat` version field selects for +formatting). + +**uv**: ✗ "Installing a tool with the same name will replace the existing tool." +Open feature request for per-directory pinning (GitHub #17813). ([uv docs]) + +**cargo**: ✗ "If the package is already installed, Cargo will reinstall it if +the installed version does not appear to be up-to-date." Only one version per +install root; use `--force` to explicitly switch versions. ([cargo docs]) + +**cargo-run-bin**: ✗ Single version per tool in `Cargo.toml` metadata. +([cargo-run-bin]) + +**npm**: ✓\* Nested `node_modules/` can contain different versions for +transitive dependencies. Package aliases allow explicit multi-version: +`npm install react15@npm:react@^15`. For CLI tools, `npx pkg@version` runs +specific versions. ([npm docs: folders]) + +#### Declarative config + +Tools can be declared in a configuration file, enabling reproducible setups. + +**Dune** (Req 4.2): `(tool)` stanza in `dune-workspace`: + +```lisp +(tool + (package + (ocamlformat + (= 0.26.2)))) +``` + +**uv**: ✗ No project-level tool configuration. Tools installed via CLI only. +([uv docs]) + +**cargo**: ✗ No built-in declarative tool configuration. Feature request was +explicitly rejected as "NOT_PLANNED" - Cargo team directed users to third-party +tools instead. ([cargo#5120]) + +**cargo-run-bin**: ✓ Declared in `Cargo.toml` metadata section: + +```toml +[package.metadata.bin] +cargo-nextest = { version = "0.9.57" } +``` + +([cargo-run-bin]) + +**npm**: ✓ Declared in `package.json` devDependencies: + +```json +{ "devDependencies": { "prettier": "^3.0.0" } } +``` + +([npm docs: package.json]) + +#### Compiler matching + +Tools are built with a compiler version matching the project to ensure ABI +compatibility. + +**Dune** (Req 3.1): Automatic by default. Tools match project compiler or system +OCaml. Opt-out via `(skip_compiler_match)`. + +**uv**: ✗ Tools ignore project-level Python version settings. "Tool environment +details: Each tool environment is linked to a specific Python version... but +will ignore non-global Python version requests like `.python-version` files and +the `requires-python` value from a `pyproject.toml`." ([uv docs]) + +**cargo**: ✓* Via rustup (not cargo itself): `rust-toolchain.toml` in the +*current directory* affects `cargo install`. Note: this uses *your project's\* +toolchain, not the tool's - a known issue ([rust-lang/cargo#11036]). ([rustup +docs]) + +[rust-lang/cargo#11036]: https://github.com/rust-lang/cargo/issues/11036 + +**cargo-run-bin**: ✗ Uses whatever Rust toolchain is active. + +**npm**: ✗ No automatic matching. Packages declare Node version via `engines` +field, but npm only warns by default (requires `.npmrc` `engine-strict=true` to +enforce). Native addons use ABI version matching via node-pre-gyp. ([npm docs: +package.json]) + +#### PATH fallback + +When a tool is not installed via the tool manager, fall back to system PATH. + +**Dune** (Req 2.4): Build rules (e.g., `dune fmt`) fall back to system PATH when +no tool is locked. Behavior for `dune tools run` is an open question (see +below). + +**uv**: ✗ Does not search system PATH for tools. + +**cargo**: ✗ `cargo install` manages `~/.cargo/bin/`, but individual commands +don't fall back to system PATH. + +**cargo-run-bin**: ✗ Only runs binaries from local `.bin/` cache. + +**npm**: ✓ npx checks `$PATH` and `node_modules/.bin` before downloading. Use +`--ignore-existing` to skip PATH lookup. (Note: "shell fallback" was a different +feature—`--shell-auto-fallback`—removed in npm v7.) ([npm v7 changes]) + +#### Precompiled binaries + +Tools can be downloaded as precompiled binaries rather than built from source. + +**Dune** (Non-req 6.5): ✗ Tools always built from source. Ensures compiler +compatibility and avoids ABI issues. + +**uv**: ✓ Downloads prebuilt wheels when available. "uv downloads and installs +packages from PyPI." ([uv docs]) + +**cargo**: ✗ `cargo install` compiles from source. ([cargo docs]) + +**cargo-run-bin**: ✓\* Optional binstall support. "Download pre-compiled +binaries using cargo-binstall." ([cargo-run-bin]) + +**npm**: ✓ Packages are prebuilt JavaScript. Native addons may require +compilation. + +#### Tool isolation + +Each tool's dependencies are isolated from other tools and from project +dependencies. + +**Dune** (Req 3.2): Each tool solved independently with its own lock directory +in `_build/.tools.lock///`. + +**uv**: ✓ "Each tool has its own isolated virtual environment." ([uv docs]) + +**cargo**: ✗ All tools share single `~/.cargo/bin/` directory. No dependency +isolation between tools. + +**cargo-run-bin**: ✓ Each tool installed to separate directory under `.bin/`. +([cargo-run-bin]) + +**npm**: ✓ Each project has isolated `node_modules/`. Tools share dependencies +within a project but are isolated between projects. + +#### Ephemeral runs + +Run a tool once without persisting its installation - useful for trying tools or +running one-off commands without polluting the workspace's tool configuration. + +**Dune** (Non-req 6.6): ✗ Tools must be locked before running. Building from +source is slow, so caching (locking) is more practical than discarding. Users +can `dune tools remove` after if needed. + +**uv**: ✓ `uvx` runs in temporary environment. "Runs a tool without installing +it... The tool environment is cached." ([uv docs: uvx]) + +**cargo**: ✗ No ephemeral execution; `cargo install` always persists. + +**cargo-run-bin**: ✗ Tools must be configured and installed first. + +**npm**: ✓ `npx` can fetch and run without adding to package.json. "Run a +command from a local or remote npm package." ([npx docs]) + +#### Batch install + +Install all declared tools with a single command. + +**Dune** (Req 4.1): `dune tools add` with no arguments locks all tools from +`(tool)` stanzas. + +**uv**: ✗ No batch install; tools installed individually via `uv tool install`. + +**cargo**: ✗ No declarative tool configuration, so no batch install. + +**cargo-run-bin**: ✓ `cargo bin --install` installs all tools from config. +"cargo bin --install - Installs all binaries." ([cargo-run-bin]) + +**npm**: ✓ `npm install` installs all devDependencies. "Install all modules +listed as dependencies in package.json." ([npm docs]) + +#### Binary selection + +When a package provides multiple executables, how to specify which one to run. + +**Dune** (Req 1.6): `--bin` flag on `dune tools run` and `dune tools path`. +Auto-selects if package provides single binary; errors with helpful message if +multiple binaries exist. Declarative via `(executable ...)` in `(tool)` stanza. + +**uv**: ✓ `--from` option specifies package when command differs from package +name: `uvx --from httpie http`. Without `--from`, uvx assumes command name +equals package name. `uv tool install` installs all binaries from a package. +([uv docs]) + +**cargo**: ✓ `--bin NAME` flag selects specific binary. Installs all binaries by +default. `default-run` in Cargo.toml only affects `cargo run`, not install. +Binary name conflicts between packages cause errors. ([cargo docs]) + +**cargo-run-bin**: ✓ `bins` array in config specifies which binaries to build: +`cargo-mobile2 = { version = "0.5.2", bins = ["cargo-android", "cargo-mobile"] }`. +Builds all binaries if `bins` omitted. ([cargo-run-bin]) + +**npm**: ✓ `--package` (or `-p`) specifies package when binary name differs: +`npx --package=foo bar`. npx auto-matches binary to package name; errors if no +match. `bin` field in package.json maps command names to scripts. ([npx docs]) + +#### Tool discovery + +How scripts and editors programmatically find paths to installed tool +executables. + +**Dune** (Req 2.2): `dune tools path ` returns the executable path for a +specific tool. Per-tool granularity enables editors to find exact binaries. + +**uv**: ✓ `uv tool dir --bin` returns the bin directory (e.g., `~/.local/bin`). +`uv tool list` shows installed tools. No per-tool path command; construct path +as `$(uv tool dir --bin)/`. ([uv docs]) + +**cargo**: ✗ No discovery command. Binaries in `~/.cargo/bin/` (or +`$CARGO_INSTALL_ROOT/bin`). `cargo install --list` shows installed crates but +not paths. Scripts use `which ` or hardcode `~/.cargo/bin/`. ([cargo +docs]) + +**cargo-run-bin**: ✗ Binaries cached in `.bin/` directory. No CLI path command; +library API (`binary::install()`) returns paths programmatically. +([cargo-run-bin]) + +**npm**: ✓ `npm bin` returns `node_modules/.bin` directory (removed in npm 9+). +npm scripts automatically get `.bin` in PATH. No per-tool path command; use +`$(npm bin)/` or `./node_modules/.bin/`. ([npm docs: bin]) + +#### Tool upgrade + +Update installed tools to newer versions, optionally preserving version +constraints. + +**Dune** (Non-req 6.2): ✗ No upgrade command. Re-run `dune tools add ` to +get latest version, or `dune tools add .` for specific version. +Constraints from `(tool)` stanza preserved on re-lock. + +**uv**: ✓ `uv tool upgrade ` or `--all`. Respects original constraints: if +installed with `black>=23,<24`, upgrade stays within that range. To change +constraints, must reinstall with `uv tool install`. ([uv docs]) + +**cargo**: ✓\* `cargo install ` auto-detects newer versions without +`--force`. No constraint preservation; always installs latest (or specified) +version. Third-party `cargo-update` provides `cargo install-update -a` for batch +upgrades. ([cargo docs]) + +**cargo-run-bin**: ✗ No upgrade command. Edit version in `Cargo.toml` manually, +then run `cargo bin --install`. Version-controlled config ensures team sync. +([cargo-run-bin]) + +**npm**: ✓ `npm update` upgrades within semver constraints (e.g., `^1.2.3` +allows minor/patch updates). `npm outdated` shows available updates. For major +version bumps beyond constraints, use third-party `npm-check-updates`. ([npm +docs: update]) + +#### System-wide install + +Install tools globally, available outside any specific project. + +**Dune**: ✗ Intentionally omitted. Tools are workspace-local only. (See +[Non-requirement 6.1](#61-system-wide-installation)) + +**uv**: ✓ `uv tool install` adds to PATH. "Executables are installed to the uv +tools directory." ([uv docs]) + +**cargo**: ✓ `cargo install` puts binaries in `~/.cargo/bin/`, typically on +PATH. ([cargo docs]) + +**cargo-run-bin**: ✗ Project-local only. + +**npm**: ✓ `npm install -g` installs globally to system. ([npm docs]) + +### Recommendations from Research + +CR-soon Alizter: Review the entire Comparison with Other Tools section for +accuracy and completeness. Verify citations are still valid and that the +comparison table reflects current design decisions. + +**Adopted in this design**: + +- **Batch install** (from cargo-run-bin, npm): `dune tools add` with no + arguments locks all tools from `(tool)` stanzas (Req 4.1) +- **Declarative config** (from npm, cargo-run-bin): `(tool)` stanza in + `dune-workspace` (Req 4.2) +- **Binary selection** (from cargo-run-bin): `(executable ...)` field and + `--bin` flag (Req 1.6) + +**Future consideration**: + +- **`--locked` flag** (from cargo-run-bin): Ensure exact versions from solver, + not just constraints +- **Ephemeral run mode** (from uv's `uvx`): See + [Non-requirement 6.6](#66-ephemeral-runs) +- **System-wide linking** (from uv): See + [Non-requirement 6.1](#61-system-wide-installation) + +[uv docs]: https://docs.astral.sh/uv/concepts/tools/ +[uv docs: uvx]: https://docs.astral.sh/uv/concepts/tools/#running-tools +[cargo docs]: https://doc.rust-lang.org/cargo/commands/cargo-install.html +[cargo#5120]: https://github.com/rust-lang/cargo/issues/5120 +[rustup docs]: https://rust-lang.github.io/rustup/overrides.html +[cargo-run-bin]: https://github.com/dustinblackman/cargo-run-bin +[npm docs]: https://docs.npmjs.com/cli/ +[npm docs: folders]: https://docs.npmjs.com/cli/v9/configuring-npm/folders +[npm docs: package.json]: + https://docs.npmjs.com/cli/v10/configuring-npm/package-json +[npm docs: bin]: https://docs.npmjs.com/cli/v8/commands/npm-bin +[npm docs: update]: https://docs.npmjs.com/cli/v10/commands/npm-update +[npx docs]: https://docs.npmjs.com/cli/v10/commands/npx +[npm v7 changes]: + https://blog.npmjs.org/post/626173315965468672/npm-v7-series-beta-release-and-semver-major.html + +## Learnings from the Prototype + +CR-soon Alizter: This entire section may be considered garbage. Extract +important information out into the spec or remove entirely. + +This section documents key insights from the initial prototype implementation. + +### What Worked Well + +1. **Versioned lock directories**: Storing tools in + `_build/.tools.lock///` cleanly separates versions and keeps + the source tree clean. The flow (solve → extract version → move to versioned + path) is straightforward. + +2. **Tool isolation via synthetic wrapper**: Solving tools with a synthetic + `_tool_wrapper` package (not mixed with project packages) prevents + "packages outside workspace" solver errors. + +3. **Install cookie as build target**: Depending on `target/cookie` rather than + individual binaries correctly handles directory targets. The cookie is + created after full installation, providing a reliable completion signal. + +4. **Disk reads for lock checks**: Reading lock directories directly from disk + (not via build system) avoids triggering expensive project builds during tool + commands. + +5. **Simple PATH-only environment**: Tools only need the bin directory in PATH, + not a full exported environment. This avoids expensive closure computations. + +6. **Unified resolution with fallback**: A single resolution chain (stanza → + locked → PATH) provides clear priority and graceful degradation. + +### Problems Discovered + +1. **Directory targets cannot be depended upon directly**: Naively depending on + `_build/.../target/bin/` fails. Solution: depend on cookie, then read + cookie to discover binaries. + +2. **Build system triggers on lock checks**: Calling `Lock_dir.get` triggered + full builds. Solution: use `Lock_dir.read_disk` for checking. + +CR-soon Alizter: Need to understand what this is about. + +3. **Checksum collection missed versioned paths**: Fetch rules initially only + collected unversioned paths. Solution: scan `.tools.lock/` for all versions. + +4. **`.pkg/` rules gated by project lock**: Initial design skipped rule + generation when no project lock existed, breaking tools-only projects. + Solution: always generate rules; create tools-only DB when needed. + +5. **Compiler matching requires explicit package**: Just constraining `ocaml` + version wasn't enough; solver picked `ocaml-base-compiler` instead of + `ocaml-system`. Solution: explicitly require `ocaml-system` when system + compiler is available. + +### Reference Packages for Testing + +The implementation should be tested against these packages. We don't +special-case them, but they represent common tool patterns and historic +dev-tools support. + +CR-soon Alizter: Vet this list. Are there other common tools to add? Any that +should be removed? Consider tools from opam-repository's most-installed +packages. + +**Historic dev-tools** (from `src/dune_pkg/dev_tool.ml`): + +| Package | Binary | Notes | +| ------------------ | ---------------- | --------------------------------- | +| `ocamlformat` | `ocamlformat` | Also has `ocamlformat-rpc` binary | +| `odoc` | `odoc` | Documentation generator | +| `ocaml-lsp-server` | `ocamllsp` | Binary name differs from package | +| `utop` | `utop` | REPL, needs project libraries | +| `earlybird` | `ocamlearlybird` | Debugger | +| `odig` | `odig` | Documentation viewer | +| `opam-publish` | `opam-publish` | Release tooling | +| `dune-release` | `dune-release` | Release tooling | +| `ocaml-index` | `ocaml-index` | Indexer for ocamllsp | +| `merlin` | `ocamlmerlin` | Editor support | + +CR-soon Alizter: These are not historic dev tools, these are dev tools that we +currently support and would like to continue to support. Linking this to the +migration requirement. They will serve as test cases sure, but they also serve +as platform roadmap integration etc. + +**Edge cases to verify**: + +- Package with single binary: auto-selection works +- Package with multiple binaries: error with helpful message +- Package name ≠ binary name: `--bin` or `(executable)` works +- Tool needing project context: `utop` with `#require` + +CR-soon Alizter: I don't like these verifications here, they are out of place + +## Ideas + +CR-someday Alizter: Consider supporting `(tool)` stanzas in `dune-project` files +for per-project tool configuration, similar to how `(pin)` exists in both +`dune-workspace` and `dune-project`. This would allow different projects within +a workspace to declare different tool requirements. Conflict resolution: +workspace wins over project, outer project wins over nested. Sibling projects +with conflicting declarations can coexist. Within project-scoped rules the +correct tool is unambiguous. CLI commands without project context (e.g., +`dune tools run` from workspace root) should error if declarations conflict. + +CR-someday Alizter: Multi-version scaling. Supporting N versions means N +separate solves, downloads, and builds. In a monorepo migrating ocamlformat +versions across 50 sub-projects, users could accumulate many versions. Consider: +(1) warning when version count exceeds threshold, (2) `dune tools gc` to remove +unused versions, (3) documenting expected steady-state (few versions, not many). + +### Editor integration + +Editors and IDEs must be able to: + +- Discover which tools are available in a workspace +- Find the path to tool executables for spawning +- Invoke tools with dune managing the process (alternative pattern) + +See [CLI commands](./implementation.md#cli-commands) for the discovery and invocation interfaces. + +**Open issues** (require consultation with editors team): + +- How editors discover which tools are available +- LSP server invocation pattern: path discovery (editor spawns process) vs + managed invocation (dune manages process). LSP servers need editor control of + stdin/stdout for JSON-RPC and process lifecycle (restart on crash). +- Single-file formatting: `dune fmt ` for editor integration ([#3244] + discusses this) + +[#3244]: https://github.com/ocaml/dune/issues/3244 + +- `dune fmt` over RPC for watch mode integration +- Integration with `.ocamlformat` version detection +- Recommended editor configuration patterns + +CR-someday Alizter: Path staleness problem. If an editor caches the result of +`dune tools path ocamllsp` and the user later updates their OCaml version, the +cached path may point to an ABI-incompatible binary. There's no mechanism to +notify editors that cached paths are invalid. Options: (1) editors always call +`dune tools path` fresh, (2) dune provides a staleness check, (3) watch mode +integration notifies editors of tool rebuilds. + diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md new file mode 100644 index 00000000000..df18839b00e --- /dev/null +++ b/doc/dev/tools-design/requirements.md @@ -0,0 +1,806 @@ +# Dune Tools Requirements + +Authors: Ali Caglayan (Tarides), Shon Feder (Tarides), Sudha Parimala (Tarides), +Ambre Suhamy (Tarides) + +## Summary + +This document specifies the requirements for dune's tool management system. + + + + +**Table of Contents** + +- [Dune Tools Requirements](#dune-tools-requirements) + - [Summary](#summary) + - [How to Read This Document](#how-to-read-this-document) + - [Terminology](#terminology) + - [Design principles](#design-principles) + - [Requirements](#requirements) + - [1. Installation](#1-installation) + - [1.1. Generality](#11-generality) + - [1.2. Scope](#12-scope) + - [1.2.1. Workspace-local scope](#121-workspace-local-scope) + - [1.2.2. Dune context scope](#122-dune-context-scope) + - [1.2.3. System wide scope](#123-system-wide-scope) + - [1.4. Version specification](#14-version-specification) + - [1.4.1. Version consistency](#141-version-consistency) + - [1.5. Clean source tree](#15-clean-source-tree) + - [1.6. Binary selection](#16-binary-selection) + - [1.7 By tool name](#17-by-tool-name) + - [1.7.1 Disambiguation](#171-disambiguation) + - [1.8 Project dependency tools](#18-project-dependency-tools) + - [1.9 Discretionary tools](#19-discretionary-tools) + - [2. Usability](#2-usability) + - [2.1 Extent of tool management ](#21-extent-of-tool-management) + - [2.1.1 when package management is enabled in a workspace](#211-when-package-management-is-enabled-in-a-workspace) + - [2.1.2 when package management is not enabled in a workspace](#212-when-package-management-is-not-enabled-in-a-workspace) + - [2.2. Shells](#22-shells) + - [2.3. Programmatic use](#23-programmatic-use) + - [2.3.1. dune subcommands](#231-dune-subcommands) + - [2.3.1.1. System PATH fallback](#2311-system-path-fallback) + - [2.3.1.1.1. When package management is enabled in a workspace](#23111-when-package-management-is-enabled-in-a-workspace) + - [2.3.1.1.2. When package management is NOT enabled in a workspace ](#23112-when-package-management-is-not-enabled-in-a-workspace) + - [2.4. Project dependency tools](#24-project-dependency-tools) + - [3. Dependency and Integration](#3-dependency-and-integration) + - [3.1 Integration axis](#31-integration-axis) + - [3.1.1. Tool isolation (I1)](#311-tool-isolation-i1) + - [3.1.1.1. Optimal builds](#3111-optimal-builds) + - [3.1.2. Tool integration (I2)](#312-tool-integration-i2) + - [3.1.2.1. Respecting integration constraints](#3121-respecting-integration-constraints) + - [4.2 Dependency axis](#42-dependency-axis) + - [4.1. Discretionary tools (D1)](#41-discretionary-tools-d1) + - [4.1.1. Not build triggers](#411-not-build-triggers) + - [4.1.2. Cannot be referenced in build rules TODO](#412-cannot-be-referenced-in-build-rules-todo) + - [4.2. Qualified dependency tools (D2) TODO](#42-qualified-dependency-tools-d2-todo) + - [4.2.1. Must be usable as tools](#421-must-be-usable-as-tools) + - [4.2.1. Must be installable via qualification](#421-must-be-installable-via-qualification) + - [4.3. Unqualified dependency tools (D3) TODO](#43-unqualified-dependency-tools-d3-todo) + - [4.3.1. Must be usable as tools TODO](#431-must-be-usable-as-tools-todo) + - [4.2.1. Must always be installed TODO](#421-must-always-be-installed-todo) + - [5. UI](#5-ui) + - [5.1. CLI](#51-cli) + - [5.1.1. Managing multiple tools](#511-managing-multiple-tools) + - [5.2. Persistent configuration for discretionary tools](#52-persistent-configuration-for-discretionary-tools) + - [5.4. Tool references in actions](#54-tool-references-in-actions) + - [5.5. Legacy migration](#55-legacy-migration) +- [TODO Orphans to put in proper section](#todo-orphans-to-put-in-proper-section) + - [-](#-) + - [4. Dog fooding](#4-dog-fooding) +- [TODO (Shon, Ambre, Sudha): Analyze out the axes of use and integration](#todo-shon-ambre-sudha-analyze-out-the-axes-of-use-and-integration) + - [There are at least 2 axes](#there-are-at-least-2-axes) + - [Usage categories spectrum:](#usage-categories-spectrum) + - [Integration requirements spectrum:](#integration-requirements-spectrum) + - [Some example](#some-example) + - [Some ideas](#some-ideas) + + + +## How to Read This Document + +[Terminology](#terminology) defines key terms used throughout the document. +Precise definition of technical terms prevents misunderstandings and certain +classes of requirement error. + +[Design principles](#design-principles) specifies the guiding principles +informing the design. + +[Requirements](#requirements) defines _what_ capabilities the system must +provide and _why_. Organized by category. Requirements describe user-facing +behaviour without specifying implementation details. Cross-references point to +the relevant Implementation sections. + +## Terminology + +- A **tool** is just an executable provided by some opam package. One opam + package can provide multiple tools. +- A **project dependency** is a component (such as a library or executable) + required for developing and/or distributing a project. For dune, this + generally means the component must be provided by a package dependency of some + package in the project. A component is a dependency if *any* of a project's + build targets depend upon it, whether or not it is correctly declared as such. +- A dependency is **qualified** when it is only required for a certain category + of build targets. Common qualification include *test* dependencies (e.g., + `alcotest`, `jq`, or `qcheck`) and *generation* dependencies (e.g., `atd`, + `menhir`, or `ocamlformat`). +- A tool is **discretionary**, relative to a particular project, if it is used + by some developers of the project, but it is not a *project dependency*. + Common examples include tools like `ocamllsp`, `utop`. +- To **install** a tool is to install the opam package providing it and making + the tool's executable available within an environment. The system responsible + for installing and uninstalling a tool is said to **manage** the tool. +- A **well-formed opam package** specifies all data necessary to install and + build its provided targets. +- A tool is **installable** if it is part of a well-formed opam package + obtainable from any source: opam repository, pinning from a source, or defined + locally. +- **Workspace** means a [dune workspace](https://dune.readthedocs.io/en/latest/explanation/scopes.html) + +## Design principles + +- **Generality**: Any executable defined in an opam packages can be a tool. +- **Orthogonality**: Dune's tool management functionality should compose with + other dune functionality with maximum flexibility and consistency, and without + introducing functional redundancies. +- **Complementarity**: Reciprocal to orthogonality, dune's tool management + functionality should integrate with and augment its core build functionality + and nascent package management functionality, and avoid introducing a + bolted-on subsystem. + +## Requirements + +NOTE: The requirements here should be compatible with the functional +requirements in +[https://ocaml.org/tools/platform-roadmap](https://ocaml.org/tools/platform-roadmap). +However, wherever the roadmap specifies implementation details, we are free to +deviate if needed to better satisfy the requirements, and our task here is not +to realize all of the requirements in the platform roadmap. + +### 1. Installation + +Users must be able to install tools via Dune. + +#### 1.1. Generality + +Any installable tool must be supported. + +##### 1.2. Scope + +The environments within which a tool are available must be scoped. + +#### 1.2.1. Workspace-local scope + +It must be possible to install tools per-workspace, such that workspaces define +a sub-environment. I.e., each workspace has its own isolated tool installations +that don't affect other workspaces or interfere with software on the host system. + +See [Directory structure](./implementation.md#directory-structure) for storage locations. + +#### 1.2.2. Dune context scope + +It must be possible to install the versions of a tool per-dune context within a +workspace, such that different contexts can use different versions of tools. + +TODO: tools as lock dirs to impl doc. In particular, specifying multiple tools +in field in one stanza, to avoid lots of duplication, but mapping this to +separate lock dirs. + +
+ +Motivation and context + + +This follows the example of `dune pkg`'s `lock_dir` in the context stanza, and +allows developers to set up different tooling configurations. + +We have sketched a preliminary design that considers `tools` stanzas as +specialized form of `lock_dir` stanza that inherits the fields of the active +`lock_dir` in a context if no overriding fields are specified. + +
+ +#### 1.2.3. System wide scope + +Users should be able to install tools in a way that allows them to be used in the +system-wide environment (e.g., outside of any particular sandbox). + +*NOTE:* This does not dictate that dune must maintain the equivalent of default +switches, or predetermine any other implementation choice. But the support for +tool management must be designed in way that makes it simple and reliable to use +installed tools outside of a project sandbox (e.g., by adding the location of a +directory of binaries in a workspace to their system `PATH`, or some other +means). + +
+ +Motivation and context + + +Related issues: + +- [dune#12107 pkg: installation of packages that can be used + system-wide](https://github.com/ocaml/dune/issues/12107) + - Even if we don't support this in the first version, the design should not + prevent this. + +
+ +#### 1.4. Version specification + +Users must be able to specify the version of tools to be installed via: + +- CLI arguments +- Declarative configuration (i.e. dune stanzas) +- Tool-specific configuration files (e.g., `.ocamlformat`) + +See [Version syntax](./implementation.md#version-syntax) for CLI syntax and +[The `(tool)` stanza](./implementation.md#the-tool-stanza) for declarative configuration. + +##### 1.4.1. Version consistency + +The versions of installed tools must remain consistent, accounting for all +configuration sources. E.g., consider an apparent conflict, such as a stanza +specifying `(= 0.26.2)` but a CLI input specifying `0.27.0`, or `.ocamlformat` says +`version=0.26.2` but the stanza says `(= 0.27.0)`: in such cases, a +consistent outcome must be derived. E.g., this could be achieved by having the +CLI input being used to update the config file, or simply by raising an error. +But it must not allow for a version to be installed that leads to inconsistent +version specifications. + +#### 1.5. Clean source tree + +Tool lock directories and built artifacts must not pollute the source tree, to +ensure that they do not inadvertently picked up in version control or otherwise +create needless noise for users. + +**Motivation**: This is a common complaint from users and is one such way to +solve the issue. Tools like `uv` handle this differently by having a global place. +Due to our compiler matching semantics it makes more sense for workspace level +and becomes fast with full caching. + +#### 1.6. Binary selection + +When a package provides multiple tools, users must be able to specify a subset +for installation. When a package providing tools is installed without +qualification, all provided tools must be installed. As a special case, when a +package provides a single binary, it will be installed without needing to qualify. + +See [The `(tool)` stanza](./implementation.md#the-tool-stanza) and [CLI +commands](./implementation.md#cli-commands) for syntax. + +**Motivation**: This is necessitated by the fact that the relation between opam +packages and tools is one-to-many: a single package can provide multiple +executables. As a result, it becomes necessary to only install a preferred +subset of the provided tools. E.g., `js_of_ocaml-compiler` provides +`js_of_ocaml`, `jsoo_minify`, and `jsoo_listunits`. + +#### 1.7 By tool name + +Users should be able to install tools based on the name of the tool without +considering the package that provides it. + +CR Shon: because this will require changes to the opam repo to be effective, +this is a *should* rather than a *must* at the moment, and may not be achievable +in the first iteration of the redisign. + +##### 1.7.1 Disambiguation + +If multiple packages provide the tools with the same name, and a user requests installation, dune should offer disambiguation. + +CR Shon: what do we do in dune package management if two packages provide the +same executable? E.g., perhaps you want to use package a for tool a' and b for +b', but they both also provide executables named `c`? + +#### 1.8 Project dependency tools + +Tools that are *project dependencies* specified as appropriately qualified +dependencies in the `dune-project` file, must be installable via installation +targets reflecting the qualification, as well as thru the build targets that +require them. + +To illustrate, this could be thru some sort of qualification to the tools +command like `dune tools install :with-test :with-dev-setup`. + +
+ +Motivation and context + + +Related issues: + +- [dune#12135 dune tools setup to install :with-dev-setup + deps](https://github.com/ocaml/dune/issues/12135) + - Should the :with-dev-setup qualifier be used by dev-tools to install? + +
+ +##### 1.9 Discretionary tools + +It must be possible to install discretionary tools without incorrectly +specifying them as if they were project dependencies. (E.g., thru a new `tool` +stanza or a CLI that updates some data stored in the workspace or `_build` +directory). + +
+ +Motivation and context + + +Related issues: + +- [dune#12913 pkg: general support for installing + tools](https://github.com/ocaml/dune/issues/12913) + - A tool can be any package with a binary. + +
+ +### 2. Usability + +Users must be able to run tools installed by Dune. + +#### 2.1 Extent of tool management + +The extent to which dune enforces management of tools within a workspace should +depend on whether or not package management is enabled in the workspace. + +##### 2.1.1 when package management is enabled in a workspace + +When users enable dune package management in a workspace, all *tools* used in the +workspace (in the precise sense defined in the [terminology](#terminology)) +should be managed dune, to the extent that dune can reasonably enforce this. +Enforcing this assumption allows dune to offer users improved guarantees about +the cohesiveness and interoperability of the provided tools. + +This does *not* entail that dune should mask or redact data from the `PATH` or +otherwise attempt to filter out a users ambient environment. But it should +provide pragmatic measures to support users by enforcing this behavior where +feasible. + +##### 2.1.2 when package management is not enabled in a workspace + +When users have not enabled dune package management in a workspace, they must be +able to use *tools* managed dune, but they should still be able to use tools +installed by opam (or other possible package managers) in all operations of +dune. + +#### 2.2. Shells + +Users must be able to run tools by invoking them directly in any shell (e.g., bash). + +See [CLI commands](./implementation.md#cli-commands) for invocation syntax. + +
+ +Motivation and context + + +Related issues: + +- [dune#12975 running dune tools exec

when p is not already installed as a dev + tool should suggest users run dune tools install

](https://github.com/ocaml/dune/issues/12975) + - The error can be something like +``` +dune tools exec merlin + Error: The tool merlin is not installed + Hint: Try 'dune tools install merlin' +``` + +

+ +#### 2.3. Programmatic use + +Programs (e.g., editor plugins) must be able to find and run installed tools via +a single, transparent mechanism. (E.g., this could be implemented by any of the +following; the `dune tools env` equivalent to `opam env` or, `dune tools path` +that prints the paths to the binaries, by adding a single directory of +executables to the lookup path, printing the path locations, or some other +means). + +See [CLI commands](./implementation.md#cli-commands) for the discovery interface. + +##### 2.3.1. dune subcommands + +As a special case, dune subcommands (e.g., `dune fmt` or `dune utop`) that +invoke external tools must be able to use tools managed by `dune tools`, when +they are available. + +###### 2.3.1.1. System PATH fallback + +The extent to which dune should allow its subcommands to fallback to the system +`PATH` when looking up required binaries depends on whether or not the workspace +has enabled package management, as dictated by +[2.1](#2-1-extent-of-tool-management). + +###### 2.3.1.1.1. When package management is enabled in a workspace + +When package management is enabled in a workspace but a required tool is not +installed, dune subcommands must produce a clear user error explaining to +users that the tool is not available and directing them to install it as a dune +managed tool. + +
+ +Motivation and context + + +Related issues: + +- [dune#10688 pkg: avoid dune fmt capturing ocamlformat from the + PATH](https://github.com/ocaml/dune/issues/10688) + - Install tools automatically? + - How would having two different package managers side by side work? (In this + case, Dune package and OPAM) + - Good to maintain compatibility with OPAM - but the tradeoff is the amount of + work needed to achieve it. + +
+ +###### 2.3.1.1.2. When package management is NOT enabled in a workspace + +When package management is enabled in a workspace and a tool is not installed +(e.g., `.ocamlformat`), dune subcommands (such as `dune fmt`, `dune build @doc`, +or `dune utop`) should fall back to the system PATH. + +**Note** This is motivated by integration with editor developers who would like +a single point of truth for running tools, and for dune to handle it. This would +mean opam users can continue to use dune in which ever way they please and the +editors will not have to care. + +#### 2.4. Project dependency tools + +When a tool is installed as a *project dependency* (under any qualification), +users must be able to execute the tool thru all the same mechanism that are +provided for executing discretionary tools. E.g., a mechanism like `dune env` +that would make the path to dune-managed binaries available must include both +discretionary tools and project dependency tools in the path. + +#### 2.4. Dog fooding + +Tools must work in the dune repository itself. Dune developers should be able to +run `dune tools add ocamlformat` and `dune tools add ocaml-lsp-server` when +working on dune. + +This is enabled by the orthogonality design (3.2): tools are solved and built +independently from project dependencies, so they don't require a working project +lock directory. + +### 3. Dependency and Integration + +Tools are depended on in numerous ways to build and develop projects and they +require different levels of integration and interdependence with the other +components (tools, libraries, or other artifacts) of a workspace. Dune must be +able to manage all installable tools across this variety, otherwise the user +experience of tool management will feel inconsistent and irregular, and users +will inevitably find certain subsets of tools unavailable or requiring +unexpected workarounds creating a fragmented and awkward user experience. + +We can further refine this requirement along two axes. + +#### 3.1 Integration axis + +Tools lie along a spectrum of integration requirements with other dependencies +in the project, which we can indicate with tree points: + +- I1: At the minimum extreme, some tools can be solved and built in complete + isolation from the rest of the project they are used in. Tools of this sort + include ocamlformat, ocp-indent, dune-release, and opam-publish. + +- I2: In the midpoint, some tools must integrate with a subset of a project's + dependencies. The most common examples in this space are tools that must + integrate with the compiler, such as ocamllsp, odoc, or merlin, but don't need + to integrate with other libraries in the workspace. Some tools also integrate + only with select components other than the compiler, such as menhir and atd. + These tools require integration with the particular version of their runtime + libraries in certain modes of use, but don't require integrations otherwise, + and since the tools themselves have a wired dependency cone than the runtime + libraries, it could sometimes be helpful to bud the executables in a separate + dependency context, pinned only to the needed runtime library version (e.g., + to avoid conflicts over a CLI parser library). + +- I3: At the maximum extreme, some tools could require be built within the + entire dependency context of the project. We are not aware of any tools that + require this currently, but we can consider utop is an illustrative example, + since its own dependencies (such as `lwt`, `xdg`, and `logs`) need to + integrate with the versions in the environment it is installed in, and it + cannot load code into the top level that would require differing on these + versions. + +Tools lying along I1 and I2 must be supported with elegant solutions where +coupling is required, and the most possible orthogonality in features when it is +not. + +##### 3.1.1. Tool isolation (I1) + +Unnecessary coupling between tools and other dependencies of a project _should_ +not be introduced. When an *installable* tool sits at I1 on the integration +axis, dune should be able to install it in a workspace, without regard to any +possible conflicts with *project dependencies*. By definition, tools at I1 have +no integration requirements with other components in a project, so there are no +grounds for such conflict. This property *should* hold rather than *must*, +because it may be infeasible and unnecessary to implement this requirement for +tools that are project dependencies. + +However, for *discretionary tools* at I1 this property *must* be guaranteed: +dune must support installation of any installable, discretionary tool at I1, +without regard to possible dependency conflicts with project dependencies. + +
+ +Motivation and context + + +To indicate just one dimension of the motivations here, consider this small set +of arbitrarily chosen packages providing I1 tools, which are all currently +published to opam and all currently incompatible with ocaml 5.5 (due to direct +or transitive constraints): + +- [cca](https://ocaml.org/p/cca/latest): "A framework for differential source + code analyses" +- [comby](https://ocaml.org/p/comby/latest): "A tool for structural code search + and replace that supports ~every language" +- [crs](https://ocaml.org/p/crs/latest): "A tool for managing inline review + comments embedded in source code" +- [depgraph](https://ocaml.org/p/depgraph/latest): "dot graphs out of ocamldep + output" +- [electrod](https://ocaml.org/p/electrod/latest): "Formal analysis for the + Electrod formal pivot language" +- [facteur](https://ocaml.org/p/facteur/latest): "Tool to send an email" +- [pfff](https://ocaml.org/p/pfff/latest): "Tools and APIs for program analysis, + code visualization, refactoring" +- [pgn_to_tex](https://ocaml.org/p/pgn_to_tex/latest): "A chess PGN to TeX conversion tool" +- [rdr](https://ocaml.org/p/rdr/latest): "Rdr is a cross-platform binary + analysis and reverse engineering tool, utilizing a unique symbol map for + global analysis." + +Without support for dependency isolation, users would be forced to chose between +avoiding recent compiler versions or making use of these (and many other) +available tools. This would violate the design principles of orthogonality and +generality, and yield and necessarily limited usability. + +See [Directory structure](./implementation.md#directory-structure) for proposed +lock directory locations. + +
+ +###### 3.1.1.1. Optimal builds + +Dune should not needless compile or rebuild dependencies that can be shared +without conflict. E.g., if the needed version of an I1 discretionary tool can be +installed by reusing the compiler version already used in the workspace, or by +pulling it from the shared cache, this should be preferred over rebuilding the +tool or its dependencies from scratch. + +##### 3.1.2. Tool integration (I2) + +When a tool requires integration with other components in the workspace at their +specific version, it must be possible to specify this requirement and have the +solution, build, and install enable that needed integration. + +Tools that integrate with the compiler are a very special case of I2 tools +requiring this need, and they have an important status among other tools because +some are used ubiquitous and because he compiler itself has a special position +in the dependency tree of any OCaml project. The most widely used tools of this +sort are ocamllsp and odoc. Dune must provide robust, intuitive, and flexible +support for managing these tools. + +
+ +Motivation and context + + +One way to address this may be thru the an equivalent of the `constraints` field +in the `lock_dir` stanza, where an absent dep-specification indicated the need +to use the same version in the context's active `lock_dir`. E.g., as + +``` +(tools + ((ocamllsp (from ocaml-lsp-server)) + utop) + (constraints ocaml)) +``` + +Which would require that the named tools be constrained on the version of ocaml +used in the workspace. + +A special purpose field could also be introduced for this purpose. + +We can also consider data added to opam files that allows this requirement to be +specified for provided tools at the package data level, instead of forcing this +upon consuming users. + +See [Compiler matching](./implementation.md#compiler-matching) for the detection +algorithm. + +
+ +###### 3.1.2.1. Respecting integration constraints + +When a user has specified the intent to install an I2 tool that is incompatible +with the required packages already installed in the workspace, dune must handle +the conflict gracefully an with clear guidance to the user. + +E.g., if a user tries to install the compiler integration `ocamllsp` and not +version is compatible with the compiler version already installed in the +workspace, dune must report and/or solve the conflict cleanly. + +This could mean automatically downgrading the compiler (if permitted by +the constraints), but a user error with clear instructions is probably just as +effectively, less surprising, and much easier to implement. E.g., "tool t cannot +be installed at version v because it requires compiler <= n, but compiler m is +currently installed. Add the following qualified constraint to your project +dependencies: `(ocaml (and (>= 5.1 (or (not :with-dev-setup) (< 5.5))))`" etc. + +Regardless of the implementation approach taken, it must be possible to +constrain the compiler versions to work with the desired integrations during +development without those constraints polluting the constraints of packages in +the workspace when they are installed. + + +#### 4.2 Dependency axis + +Tools lie along a spectrum of dependency status (i.e., to what extent they are +dependencies for the project) which we can indicate with these tree points: + +- D1. At the minimum extreme, *discretionary tools* are not project dependencies + at all. Rather, they are tools we want to install and run on an ad hoc basis, + for use as developers. E.g., `ocamllsp` or `ocamlgrep`. +- D2. In the midpoint, *qualified project dependency* tools are used in rules associated + with a build alias, but not as part of the required package dependencies to + install any packages. Example include any tools that are qualified in + opam packages with the `with-test`, or `with-doc` filters. +- D3. At the maximum extreme, unqualified *project dependency* tools are + required for an installation build of packages in the project, and need to be + specified as package dependencies unconditionally.E.g., `menhir` or `atd` when + used as part of the project build (by contrast, if these are used only for code + generation, they could be qualified project dependencies behind a hypothetical + `with-gen` filter). + +Note that, since tools like `utop`, `odoc`, and `ocamlformat` have dedicated +builtin rules from dune, they are technically not D1. + +##### 4.1. Discretionary tools (D1) + +Dune must be able to manage discretionary tools. + +###### 4.1.1. Not build triggers + +Locking or adding discretionary tools must not trigger project builds. + +###### 4.1.2. Cannot be referenced in build rules + +Discretionary tools managed by dune must not be referenced in user build rules: dune +must not add these binaries to its path when running rules, and must not +resolve `%{bin:...}` forms that attempt to reference them. + +By definition, tools referenced in rules are at least D2. + +###### 4.1.2.1. Useful guidance on invalid reference in rules + +When a discretionary tool is configured and an invalid reference to it is found +in a build rule, dune should report an error with clear guidance to users, +advising them to move the tool configuration into the appropriate package +dependency. + +###### 4.1.3. Subset of D2 and D3 tools functionality + +The functionality of discretionary, D1 tools must a strict subset of the functionality over +D2 and D3 project dependency tools: anything you can do with + +##### 4.2. Qualified project dependency tools (D2) + +Qualified project dependency tools must be installable based on the existing +package dependency configuration mechanism. This configuration may be extended +to support tool-specific configuration when the need and means to achieve that +is known. + +
+ +Motivation and context + + +When tools are invoked in build rules, they are dependencies of a project, since +the full breadth of the project cannot be built without installing the tool. We +have an existing mechanism for declaring dependencies for installation, and so +based on our principles of orthogonality and complementarity, we must not +introduce redundant ways of solving the same problem that is already accounted +for. Doing so would not only be inelegant, but lead to user confusion and +package specification fragmentation, since the existing systems is also +supported by opam. + +
+ +###### 4.2.1. Usable as tools + +Qualified project dependency tools must be usable in exactly the same ways as +discretionary, D1 tools. E.g., if D1 tools can be run via a command like `dune +tools exec ...` then so too must be D2 tools. However, the implementation is +not prescribed by the requirement, and we could instead permit D1 tools to be +run via the existing `dune exec ...` subcommand, providing a simpler and more +intuitive interface to users than requiring to separate subcommands. + +###### 4.2.2. Installable via qualification + +If a set of tools are qualified with a filter such as `:with-test` or +`:with-doc`, it must be possible to install just that set together (in addition +to the unqualified dependencies). + +###### 4.2.3. Builtin D2 tools + +A select subset of keystone tools are treated by dune as builtin qualified +dependencies, including `odoc`, `utop`, and `ocamlformat` as the most widely +used. These should be treated as if they have qualified dependencies built in, +with further constraint or specification of how to install them available as a +user override on top of the default configuration. + +##### 4.3. Unqualified dependency tools (D3) TODO + +###### 4.3.1. Usable as tools + +As with 4.2.1, unqualified project dependency tools must be usable in exactly +the same ways as D1 and D2 tools. + +###### 4.2.1. Installed on build + +Unqualified tools must be installed when a project is built. + +(Already satisfied by the current implementation of package managemet.) + +#### 5. UI + +#### 5.1. CLI + +Users must be able to manage tools using CLI commands: + +- Add/lock individual tools to the workspace +- Run tools (building if needed) +- List locked tools and versions +- Remove tools +- Discover paths to tool executables + +TODO: we may want a req. that `dune clean` does not wipe out tools + +CLI-added tools persist until `dune clean` or `dune tools remove`, but are not +reproducible across clean builds. See [Batch +operations](./implementation.md#batch-operations) for batch commands. + +##### 5.1.1. Managing multiple tools + +It must be possible to change (e.g., update, install, or remove) the +installation of a set of configured tools by issuing a single command. + +
+ +Motivation and context + + +We must not require users to do the tedious work of running the same command +over and over to install a set of tools. How this is addressed is left as an +implementation detail. E.g., it could mean supporting `dune tools install a b c` +or just having a command that installs all configured tools (see 5.2) in one +command. + +Related issues: + +- [dune#12557 dune tools install should take multiple package + arguments](https://github.com/ocaml/dune/issues/12557) + - Discussed in the implementation spec. See [./implementation.md#batch-operations](./implementation.md#batch-operations) + +
+ +##### 5.1.2. Watch mode integration + +Tool operations (e.g., `dune tools add`, `run`, etc.) must work correctly when a +watch server is running (`dune build -w`). Rather than directly manipulating +lock directories, tool commands should coordinate with the watch server via RPC +to avoid races and ensure the server picks up newly added tools. + +##### 5.1.3 Avoid invocation collisions + +When a `dune tools` command is run that depends on the state of the workspace, +it should not interfere with concurrent running dune commands or lead to invalid +results or states. This may just mean refusing to run if the build directory is +locked, cooperating to sequence requests, or something else. + +#### 5.2. Persistent configuration for discretionary tools + +Users must be able to declare discretionary tools in the workspace +configuration. + +
+ +Motivation and context + + +This allows users to share tooling configurations between users and across fresh +project setups as part of a reusable ad hoc dev tool setup. + +- Declaration is version-controlled and shared with collaborators +- CI and fresh checkouts get the same tools + +See [The `(tool)` stanza](./implementation.md#the-tool-stanza) for proposed syntax. + +
+