From 609a10260f8b5cfb36f6909a41c9fa5a1f07967a Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Sun, 25 Jan 2026 22:31:55 -0500 Subject: [PATCH 01/72] Add initial draft of tool management design Signed-off-by: Shon Feder --- doc/dev/tools.md | 131 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 doc/dev/tools.md diff --git a/doc/dev/tools.md b/doc/dev/tools.md new file mode 100644 index 00000000000..75ef29d0c7c --- /dev/null +++ b/doc/dev/tools.md @@ -0,0 +1,131 @@ +# Tools + +This document sets out the design of dune's support for managing tools (e.g., +dev tools, such as ocamlformat or odig). The support for managing +tools is implemented on the basis of dune package management, but we can and +should make changes to the implementation of the package management system, +when needed to achieve robust and usable tool management. + + +**Table of Contents** + +- [Tools](#tools) + - [Terminology](#terminology) + - [Requirements](#requirements) + - [1. Installation](#1-installation) + - [1.1. Generality](#11-generality) + - [1.3. Project sandbox](#13-project-sandbox) + - [1.4. System wide](#14-system-wide) + - [1.5. Dog fooding](#15-dog-fooding) + - [2. Usability](#2-usability) + - [2.1. Shells](#21-shells) + - [2.2 Programmatic use](#22-programmatic-use) + - [2.3. Dune subcommands](#23-dune-subcommands) + - [2.4. Dog fooding](#24-dog-fooding) + - [3. Dependency interactions](#3-dependency-interactions) + - [3.1. Constrained dependencies](#31-constrained-dependencies) + - [3.2. Independence](#32-independence) + - [4. UI](#4-ui) + - [4.1. CLI](#41-cli) + - [4.2. Persistent configuration](#42-persistent-configuration) + - [TODO: Comparison with other tools](#todo-comparison-with-other-tools) + - [TODO: Learnings from the initial dune prototype](#todo-learnings-from-the-initial-dune-prototype) + + + +## Terminology + +- A **tool** is just an executable provided by some opam package. +- A tool is **available** if it is part of a well formed opam package published + on the opam repository, or via pinning from any source (e.g., a Codeberg + repository). + +## Requirements + +NOTE: The requirements here should be compatible with the functional +requirements in +[https://ocaml.org/tools/platform-roadmap](https://ocaml.org/tools/platform-roadmap). +However, wherever the roadmap specifies implementation details, we are free to +deviate if needed to better satisfy the requirements. + +### 1. Installation +Users must be able to install tools by Dune. + +#### 1.1. Generality +All available tools must be installable, without requiring changes to the dune +source code. + +#### 1.3. Project sandbox +Users must be able to install tools within the scope of a project sandbox (i.e., +a "workspace"). + +#### 1.4. System wide +Users must be able to install tools in a way that allows them to be used in the +system-wide environment (e.g., outside of any particular sandbox). + +*NOTE:* This does not dictate that dune must maintain the equivalent of +default switches, or predetermine any other implementation choice. But the +support for tool management must be designed in way that makes it simple and +reliable for users to use installed tools outside of a project sandbox (e.g., by +adding the location of a directory of binaries to their `PATH` or some other +means). + +#### 1.5. Dog fooding +When projects include the implementation of tools, the tools must installable +from the project source. + +### 2. Usability +Users must be able to run tools installed by Dune. + +#### 2.1. Shells +Users must be able to run tools by invoking them directly in a shell (e.g., bash). + +#### 2.2 Programmatic use +Other programs (e.g., editor plugins) must be able to find and run installed +tools via a single, transparent mechanism (e.g., an equivalent to `opam env` or +by adding a single directory of executables to the lookup path, or some other +means). + +#### 2.3. Dune subcommands +When dune subcommands make use of tools (e.g., `dune fmt`), they must be able to +use the tools installed by Dune. + +#### 2.4. Dog fooding +The tools installable from a project's source must be usable within the project +where they are implemented. + +### 3. Dependency interactions +Tools installed by dune should have the minimal necessary interaction with +other dependencies of the environment they are installed in. + +#### 3.1. Constrained dependencies +When a tool requires constraints on the version of other tools used in the same +context (e.g, ocamllsp and the ocaml compiler), this constraint must be +detected and enforced, to ensure a coherent environment. + +This should be enforceable and configurable for any tools with such constraints. + +#### 3.2. Independence +Any dependencies of a tool that are not constrained in the way described in +*3.1* should be installed in a separate context, to avoid unnecessary +interference with the dependencies of the rest of the context. + +### 4. UI + +#### 4.1. CLI + +Users must be able to manage tools using the CLI interface. + +#### 4.2. Persistent configuration + +When users wish to share a set of installed tools, or a particular tooling +configuration, they must be able to do so via configuration data in a file +(e.g., stanzas in a `dune-workspace` etc.). + +## TODO: Comparison with other tools + +- uv +- cargo +- ?? + +## TODO: Learnings from the initial dune prototype From b03b1a53fb5c04f0de21d83258acfa49de7076c5 Mon Sep 17 00:00:00 2001 From: Ali Caglayan Date: Wed, 4 Mar 2026 12:12:51 +0100 Subject: [PATCH 02/72] update opening paragraph Signed-off-by: Ali Caglayan --- doc/dev/tools.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 75ef29d0c7c..d9f5ee1ee6b 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -1,10 +1,9 @@ # Tools This document sets out the design of dune's support for managing tools (e.g., -dev tools, such as ocamlformat or odig). The support for managing -tools is implemented on the basis of dune package management, but we can and -should make changes to the implementation of the package management system, -when needed to achieve robust and usable tool management. +dev tools, such as ocamlformat or odig). Tools are configured via a `(tool)` +stanza in `dune-workspace` or via the CLI, and any opam package that provides +executables can be used as a tool. **Table of Contents** From 0f63fd80dbada2920dc285332c174168828f19ca Mon Sep 17 00:00:00 2001 From: Ali Caglayan Date: Wed, 4 Mar 2026 12:24:22 +0100 Subject: [PATCH 03/72] remove terminology section Let's add this back when the document takes shape to clarify confusion. At the moment it is not adding anything and is potentially confusing. Signed-off-by: Ali Caglayan --- doc/dev/tools.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index d9f5ee1ee6b..6fe53754924 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -9,7 +9,6 @@ executables can be used as a tool. **Table of Contents** - [Tools](#tools) - - [Terminology](#terminology) - [Requirements](#requirements) - [1. Installation](#1-installation) - [1.1. Generality](#11-generality) @@ -32,13 +31,6 @@ executables can be used as a tool. -## Terminology - -- A **tool** is just an executable provided by some opam package. -- A tool is **available** if it is part of a well formed opam package published - on the opam repository, or via pinning from any source (e.g., a Codeberg - repository). - ## Requirements NOTE: The requirements here should be compatible with the functional From 68bb3f918f3205c5315fea1633c940591b39645a Mon Sep 17 00:00:00 2001 From: Ali Caglayan Date: Wed, 4 Mar 2026 12:26:08 +0100 Subject: [PATCH 04/72] fix numbering Signed-off-by: Ali Caglayan --- doc/dev/tools.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 6fe53754924..cb42a01f98e 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -12,9 +12,9 @@ executables can be used as a tool. - [Requirements](#requirements) - [1. Installation](#1-installation) - [1.1. Generality](#11-generality) - - [1.3. Project sandbox](#13-project-sandbox) - - [1.4. System wide](#14-system-wide) - - [1.5. Dog fooding](#15-dog-fooding) + - [1.2. Project sandbox](#12-project-sandbox) + - [1.3. System wide](#13-system-wide) + - [1.4. Dog fooding](#14-dog-fooding) - [2. Usability](#2-usability) - [2.1. Shells](#21-shells) - [2.2 Programmatic use](#22-programmatic-use) @@ -46,11 +46,11 @@ Users must be able to install tools by Dune. All available tools must be installable, without requiring changes to the dune source code. -#### 1.3. Project sandbox -Users must be able to install tools within the scope of a project sandbox (i.e., +#### 1.2. Project sandbox +Users must be able to install tools within the scope of a project sandbox (i.e., a "workspace"). -#### 1.4. System wide +#### 1.3. System wide Users must be able to install tools in a way that allows them to be used in the system-wide environment (e.g., outside of any particular sandbox). @@ -61,7 +61,7 @@ reliable for users to use installed tools outside of a project sandbox (e.g., by adding the location of a directory of binaries to their `PATH` or some other means). -#### 1.5. Dog fooding +#### 1.4. Dog fooding When projects include the implementation of tools, the tools must installable from the project source. From e7bb12859e2f5795e60ac3c1eb7f23c97b256386 Mon Sep 17 00:00:00 2001 From: Ali Caglayan Date: Wed, 4 Mar 2026 12:34:06 +0100 Subject: [PATCH 05/72] update requirements, issues and research Signed-off-by: Ali Caglayan --- doc/dev/tools.md | 2023 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 1959 insertions(+), 64 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index cb42a01f98e..c28fbc3eb60 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -1,33 +1,174 @@ # Tools -This document sets out the design of dune's support for managing tools (e.g., -dev tools, such as ocamlformat or odig). Tools are configured via a `(tool)` -stanza in `dune-workspace` or via the CLI, and any opam package that provides -executables can be used as a tool. +Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) + +This document specifies dune's tool management system for development tools like +ocamlformat, ocamllsp, and odoc. + +## Summary + +Tools are executables you use during development but that aren't part of your +project's build output. Think of each tool as an invisible opam switch: +independently solved, with its own dependency closure, isolated from your +project. + +**Why a new system?** The legacy dev-tools system only supported a hardcoded +list of tools and required dune package management. The new system works with +any opam package, whether you use dune's package management, an opam switch, or +system OCaml. + +**Design principles:** + +- **Generality**: Any opam package providing executables can be a tool +- **Orthogonality**: Tools are solved independently from project dependencies +- **Clean source tree**: All tool artifacts live in `_build/`, not your repo +- **Multi-version**: Different projects can use different tool versions + +## How to Read This Document + +**Requirements** defines _what_ capabilities the system must provide and _why_. +Organized by category (installation, usability, dependencies, UI). Requirements +describe user-facing behavior without specifying syntax or implementation. +Cross-references point to the relevant Specification sections. + +**Non-requirements** explicitly lists what's out of scope and why. + +**Specification** details _how_ the system implements the requirements: stanza +syntax, CLI commands, version resolution algorithm, and directory structure. +Each section notes which requirements it implements. Open questions are marked +where decisions are pending. + +**Relationship to Package Management** explains the orthogonality principle and +how tools differ from project dependencies. + +**Comparison with Other Tools** analyzes how uv, cargo, cargo-run-bin, and npm +handle tool management, informing our design decisions. + +**CR markers** indicate areas needing further work: CR-soon for items to address +before stable release, CR-someday for post-v1 considerations. + + **Table of Contents** - [Tools](#tools) + - [Summary](#summary) + - [How to Read This Document](#how-to-read-this-document) - [Requirements](#requirements) + - [Terminology](#terminology) - [1. Installation](#1-installation) - [1.1. Generality](#11-generality) - - [1.2. Project sandbox](#12-project-sandbox) - - [1.3. System wide](#13-system-wide) - - [1.4. Dog fooding](#14-dog-fooding) + - [1.2. Workspace-local](#12-workspace-local) + - [1.3. Version pinning](#13-version-pinning) + - [1.4. Multi-version support](#14-multi-version-support) + - [1.5. Clean source tree](#15-clean-source-tree) + - [1.6. Binary selection](#16-binary-selection) - [2. Usability](#2-usability) - - [2.1. Shells](#21-shells) - - [2.2 Programmatic use](#22-programmatic-use) + - [2.1. Tool invocation](#21-tool-invocation) + - [2.2. Programmatic use](#22-programmatic-use) - [2.3. Dune subcommands](#23-dune-subcommands) - - [2.4. Dog fooding](#24-dog-fooding) + - [2.4. System PATH fallback](#24-system-path-fallback) + - [2.5. Editor integration](#25-editor-integration) - [3. Dependency interactions](#3-dependency-interactions) - - [3.1. Constrained dependencies](#31-constrained-dependencies) - - [3.2. Independence](#32-independence) + - [3.1. Compiler compatibility](#31-compiler-compatibility) + - [3.2. Dependency isolation](#32-dependency-isolation) + - [3.3. No build triggers](#33-no-build-triggers) + - [3.4. Watch mode integration](#34-watch-mode-integration) + - [3.5. Dog fooding](#35-dog-fooding) - [4. UI](#4-ui) - [4.1. CLI](#41-cli) - [4.2. Persistent configuration](#42-persistent-configuration) - - [TODO: Comparison with other tools](#todo-comparison-with-other-tools) - - [TODO: Learnings from the initial dune prototype](#todo-learnings-from-the-initial-dune-prototype) + - [5. Dune Integration](#5-dune-integration) + - [5.1. Format rules (`dune fmt`, `dune build @fmt`)](#51-format-rules-dune-fmt-dune-build-fmt) + - [5.2. Documentation rules (`dune build @doc`, `dune ocaml doc`)](#52-documentation-rules-dune-build-doc-dune-ocaml-doc) + - [5.3. REPL (`dune utop`, `dune ocaml utop`)](#53-repl-dune-utop-dune-ocaml-utop) + - [5.4. Tool references in actions](#54-tool-references-in-actions) + - [5.5. Legacy migration](#55-legacy-migration) + - [6. Non-requirements](#6-non-requirements) + - [6.1. System-wide installation](#61-system-wide-installation) + - [6.2. Automatic updates](#62-automatic-updates) + - [6.3. Tools for tools](#63-tools-for-tools) + - [6.4. Environment manipulation](#64-environment-manipulation) + - [6.5. Precompiled binaries](#65-precompiled-binaries) + - [6.6. Ephemeral runs](#66-ephemeral-runs) + - [Specification](#specification) + - [The `(tool)` stanza](#the-tool-stanza) + - [Syntax](#syntax) + - [Fields](#fields) + - [Compiler matching](#compiler-matching) + - [Examples](#examples) + - [Invalid tool selections](#invalid-tool-selections) + - [CLI commands](#cli-commands) + - [Version syntax](#version-syntax) + - [Batch operations](#batch-operations) + - [Open question: version separator](#open-question-version-separator) + - [Open question: `add` vs `install` naming](#open-question-add-vs-install-naming) + - [Open question: `dune tools run` PATH fallback](#open-question-dune-tools-run-path-fallback) + - [Tool resolution](#tool-resolution) + - [Source resolution](#source-resolution) + - [Version selection](#version-selection) + - [Example scenarios](#example-scenarios) + - [Directory structure](#directory-structure) + - [Tool pforms](#tool-pforms) + - [Syntax](#syntax-1) + - [Resolution](#resolution) + - [Examples](#examples-1) + - [Future: library access](#future-library-access) + - [Relationship to Package Management](#relationship-to-package-management) + - [Orthogonality: a key design principle](#orthogonality-a-key-design-principle) + - [The package-tool continuum](#the-package-tool-continuum) + - [Key differences from project dependencies](#key-differences-from-project-dependencies) + - [Why tools don't use `dune.lock/`](#why-tools-dont-use-dunelock) + - [Locking vs building](#locking-vs-building) + - [Future: Relocatable compiler and caching](#future-relocatable-compiler-and-caching) + - [Potential future stanza options](#potential-future-stanza-options) + - [GitHub Issues Addressed](#github-issues-addressed) + - [Arbitrary tool packages ([#12913])](#arbitrary-tool-packages-12913) + - [Lock directory clutter ([#10955], [#12097])](#lock-directory-clutter-10955-12097) + - [Compiler ABI mismatch ([#11229])](#compiler-abi-mismatch-11229) + - [Graceful fallback for missing tools ([#10578])](#graceful-fallback-for-missing-tools-10578) + - [Version constraints ([#12777])](#version-constraints-12777) + - [Version consistency ([#10688])](#version-consistency-10688) + - [Dev tools redesign ([#12914])](#dev-tools-redesign-12914) + - [Tool dependency isolation ([#12551])](#tool-dependency-isolation-12551) + - [No project build triggers ([#11037])](#no-project-build-triggers-11037) + - [Compiler-independent formatters ([#11038])](#compiler-independent-formatters-11038) + - [Version from config files ([#5315])](#version-from-config-files-5315) + - [Version constraint parsing ([#12866])](#version-constraint-parsing-12866) + - [Opam with-dev-setup integration ([#12135])](#opam-with-dev-setup-integration-12135) + - [Helpful error messages ([#13235], [#12975])](#helpful-error-messages-13235-12975) + - [Batch installation ([#12557])](#batch-installation-12557) + - [Failure isolation ([#12818])](#failure-isolation-12818) + - [Utop findlib integration ([#13471])](#utop-findlib-integration-13471) + - [Comparison with Other Tools](#comparison-with-other-tools) + - [uv (Python)](#uv-python) + - [cargo (Rust)](#cargo-rust) + - [cargo-run-bin (Rust community tool)](#cargo-run-bin-rust-community-tool) + - [npm (JavaScript)](#npm-javascript) + - [How Our Design Compares](#how-our-design-compares) + - [Tools compared](#tools-compared) + - [Comparison table](#comparison-table) + - [Project-local](#project-local) + - [Multi-version](#multi-version) + - [Declarative config](#declarative-config) + - [Compiler matching](#compiler-matching-1) + - [PATH fallback](#path-fallback) + - [Precompiled binaries](#precompiled-binaries) + - [Tool isolation](#tool-isolation) + - [Ephemeral runs](#ephemeral-runs) + - [Batch install](#batch-install) + - [Binary selection](#binary-selection) + - [Tool discovery](#tool-discovery) + - [Tool upgrade](#tool-upgrade) + - [System-wide install](#system-wide-install) + - [Recommendations from Research](#recommendations-from-research) + - [Learnings from the Prototype](#learnings-from-the-prototype) + - [What Worked Well](#what-worked-well) + - [Problems Discovered](#problems-discovered) + - [Reference Packages for Testing](#reference-packages-for-testing) @@ -39,84 +180,1838 @@ requirements in However, wherever the roadmap specifies implementation details, we are free to deviate if needed to better satisfy the requirements. +### Terminology + +**Tool**: An executable provided by an opam package, used during development but +not part of the project's build output. Examples: ocamlformat (formatter), +ocamllsp (language server), odoc (documentation generator), utop (REPL). + +**Install**: Make a tool available for execution in the workspace. Unlike global +installation (`opam install`), tools are workspace-local. See +[CLI commands](#cli-commands) and [Directory structure](#directory-structure). + +CR-soon Alizter: Ephemeral could be explained here, its a bit obscure + ### 1. Installation -Users must be able to install tools by Dune. + +Users must be able to install tools via Dune. #### 1.1. Generality -All available tools must be installable, without requiring changes to the dune -source code. -#### 1.2. Project sandbox -Users must be able to install tools within the scope of a project sandbox (i.e., -a "workspace"). +Any opam package that provides executables can be used as a tool. Unlike the +legacy system, tools are not hardcoded into dune. Users can install arbitrary +packages without waiting for dune releases. + +#### 1.2. Workspace-local + +Tools are installed per-workspace, not globally. Each workspace has its own +isolated tool installations that don't affect other workspaces or the system. + +See [Directory structure](#directory-structure) for storage locations. + +CR-soon Alizter: Once we have the relocatable compiler, tools being built will +be cached so they aren't "installed". They will materialise in \_build/ but that +shouldn't be a persistent store. + +CR-someday Alizter: Consider supporting `(tool)` stanzas in `dune-project` files +for per-project tool configuration, similar to how `(pin)` exists in both +`dune-workspace` and `dune-project`. This would allow different projects within +a workspace to declare different tool requirements. Conflict resolution: +workspace wins over project, outer project wins over nested. Sibling projects +with conflicting declarations can coexist. Within project-scoped rules the +correct tool is unambiguous. CLI commands without project context (e.g., +`dune tools run` from workspace root) should error if declarations conflict. + +#### 1.3. Version pinning + +Users must be able to install specific versions of tools via: + +- CLI arguments +- Declarative configuration +- Tool-specific configuration files (e.g., `.ocamlformat`) + +See [Version syntax](#version-syntax) for CLI syntax and +[The `(tool)` stanza](#the-tool-stanza) for declarative configuration. + +CR-soon Alizter: Edge cases to specify: + +- Stanza says `(= 0.26.2)` but CLI requests `0.27.0`: which wins? +- `.ocamlformat` says `version=0.26.2` but stanza says `(= 0.27.0)`: conflict? +- Version doesn't exist in opam-repository: error message? + +#### 1.4. Multi-version support + +Multiple versions of the same tool can coexist within a workspace. This enables +per-project tool resolution - for example, different projects within a workspace +can have different `.ocamlformat` files specifying different versions, and the +formatting rules will use the correct version for each project. -#### 1.3. System wide -Users must be able to install tools in a way that allows them to be used in the -system-wide environment (e.g., outside of any particular sandbox). +CR-someday Alizter: Multi-version scaling. Supporting N versions means N +separate solves, downloads, and builds. In a monorepo migrating ocamlformat +versions across 50 sub-projects, users could accumulate many versions. Consider: +(1) warning when version count exceeds threshold, (2) `dune tools gc` to remove +unused versions, (3) documenting expected steady-state (few versions, not many). -*NOTE:* This does not dictate that dune must maintain the equivalent of -default switches, or predetermine any other implementation choice. But the -support for tool management must be designed in way that makes it simple and -reliable for users to use installed tools outside of a project sandbox (e.g., by -adding the location of a directory of binaries to their `PATH` or some other -means). +See [Tool pforms](#tool-pforms) for how build rules can reference versioned tool +executables. -#### 1.4. Dog fooding -When projects include the implementation of tools, the tools must installable -from the project source. +#### 1.5. Clean source tree + +Tool lock directories and built artifacts must not pollute the source tree. They +should be stored in build output directories that are: + +- Excluded from version control +- Cleaned by standard build cleanup +- Invisible to users during normal development + +See [Directory structure](#directory-structure) for exact paths. + +CR-soon Alizter: This is a common complaint from users and is one such way to +solve the issue. Tools like uv handle this differently by having a global place. +Due to our compiler matching semantics it makes more sense for workspace level +and becomes fast with full caching. + +#### 1.6. Binary selection + +When a tool package provides multiple executables, users must be able to: + +- Specify which binary to run (per-invocation) +- Configure a default binary (persistent) + +When a package provides a single binary, it should be selected automatically. + +See [The `(tool)` stanza](#the-tool-stanza) and [CLI commands](#cli-commands) +for syntax. + +CR-soon Alizter: Here is an annoyance: opam packages don't specify which +binaries they install, which is fine. Once we have built the package we can +inspect which binaries we have. For something like `ocaml-lsp-server` this will +be the `ocamllsp` binary. For `ocamlformat` there are two: `ocamlformat` which +is intended and `ocamlformat-rpc` which is likely not, in this case the +prototype errors and asks the user via the CLI to be more specific by providing +the `--bin` arg. We should evaluate if this is going to be the best way to solve +this issue. ### 2. Usability + Users must be able to run tools installed by Dune. -#### 2.1. Shells -Users must be able to run tools by invoking them directly in a shell (e.g., bash). +#### 2.1. Tool invocation + +Users must be able to: + +- Run installed tools from any shell +- Request a specific version when multiple versions are installed + +See [CLI commands](#cli-commands) for invocation syntax. + +CR-soon Alizter: Edge cases to specify: + +- Tool not locked: error with suggestion to run `dune tools add`? +- Multiple versions locked, none specified: error listing available versions? +- Specified version not locked: error with suggestion? +- Build fails: propagate build error? -#### 2.2 Programmatic use -Other programs (e.g., editor plugins) must be able to find and run installed -tools via a single, transparent mechanism (e.g., an equivalent to `opam env` or -by adding a single directory of executables to the lookup path, or some other -means). +#### 2.2. Programmatic use + +Other programs (e.g., editor plugins, scripts) must be able to programmatically +discover the path to a tool's executable without parsing internal directory +structures. + +See [CLI commands](#cli-commands) for the discovery interface. + +CR-soon Alizter: To be clear, this is replacing the `dune tools which ` +that we currently have. Not sure if its the best design. + +CR-soon Alizter: `dune tools path` behavior is underspecified: + +- Should `path` trigger download and build if not yet built? Or only return path + if already built, erroring otherwise? +- If `--bin` is specified, can we compute the path without building (since path + is deterministic from package/version/binary name)? +- If package has multiple binaries and no `--bin`: error immediately, or + download and build first to discover available binaries? +- Tool not locked: error, or fall back to `which`? +- How does `path` interact with `(tool)` stanzas vs CLI-added tools? Same + resolution as `run`? + +CR-someday Alizter: Consider a single bin directory with symlinks to all tool +executables (like npm's `node_modules/.bin/`). Instead of per-tool paths, have +`_build/.tools/bin/` containing symlinks to all installed tool binaries. This +simplifies editor integration (one directory to add to PATH), avoids PATH length +limits (especially on Windows), and provides a single stable location for +discovery. Trade-off: need to maintain symlinks as tools are added/removed. #### 2.3. Dune subcommands -When dune subcommands make use of tools (e.g., `dune fmt`), they must be able to -use the tools installed by Dune. -#### 2.4. Dog fooding -The tools installable from a project's source must be usable within the project -where they are implemented. +Dune subcommands that invoke external tools must be able to use tools managed by +`dune tools`. See [Dune Integration](#5-dune-integration). + +#### 2.4. System PATH fallback + +When a tool is not locked and no version is specified (e.g., `.ocamlformat` +without a version), dune subcommands should fall back to the system PATH. + +**Note** This is motivated by integration with editor developers who would like +a single point of truth for running tools, and for dune to handle it. This would +mean opam users can continue to use dune in which ever way they please and the +editors will not have to care. + +CR-Alizter soon: Fixup wording above. + +CR-Alizter soon: How would this even work if we don't know the binary from the +package? + +#### 2.5. Editor integration + +Editors and IDEs must be able to: + +- Discover which tools are available in a workspace +- Find the path to tool executables for spawning +- Invoke tools with dune managing the process (alternative pattern) + +See [CLI commands](#cli-commands) for the discovery and invocation interfaces. + +**Open issues** (require consultation with editors team): + +- How editors discover which tools are available +- LSP server invocation pattern: path discovery (editor spawns process) vs + managed invocation (dune manages process). LSP servers need editor control of + stdin/stdout for JSON-RPC and process lifecycle (restart on crash). +- Single-file formatting: `dune fmt ` for editor integration ([#3244] + discusses this) + +[#3244]: https://github.com/ocaml/dune/issues/3244 + +- `dune fmt` over RPC for watch mode integration +- Integration with `.ocamlformat` version detection +- Recommended editor configuration patterns + +CR-someday Alizter: Path staleness problem. If an editor caches the result of +`dune tools path ocamllsp` and the user later updates their OCaml version, the +cached path may point to an ABI-incompatible binary. There's no mechanism to +notify editors that cached paths are invalid. Options: (1) editors always call +`dune tools path` fresh, (2) dune provides a staleness check, (3) watch mode +integration notifies editors of tool rebuilds. ### 3. Dependency interactions -Tools installed by dune should have the minimal necessary interaction with -other dependencies of the environment they are installed in. -#### 3.1. Constrained dependencies -When a tool requires constraints on the version of other tools used in the same -context (e.g, ocamllsp and the ocaml compiler), this constraint must be -detected and enforced, to ensure a coherent environment. +Tools installed by dune should have the minimal necessary interaction with other +dependencies of the environment they are installed in. + +#### 3.1. Compiler compatibility + +By default, tools are built with a compiler matching the environment. This +ensures tools like ocamllsp can read project build artifacts correctly. + +Users should be able to opt out of compiler matching per-tool for tools that +don't need it (e.g., formatters that only parse source text). + +CR-soon Alizter: The utility of opting out seems debatable, but is part of the +package-tool continuum approach. + +See [Compiler matching](#compiler-matching) for the detection algorithm. + +CR-someday Alizter: Reverse influence - could tools constrain the project? E.g., +if ocamllsp isn't available for OCaml 5.3 yet, a user might want their tool +requirements to influence which compiler version they use. Currently tools are +fully isolated and don't affect project solving. This would be a significant +departure from the orthogonality principle. + +#### 3.2. Dependency isolation + +Each tool is solved independently with its own lock directory. Tool dependencies +do not affect the project's `dune.lock`, and vice versa. + +See [Directory structure](#directory-structure) for lock directory locations. + +#### 3.3. No build triggers + +Locking or adding tools must not trigger project builds. + +Build rules that _use_ tools (e.g., formatting) are a separate concern; see +[Dune Integration](#5-dune-integration). -This should be enforceable and configurable for any tools with such constraints. +#### 3.4. Watch mode integration -#### 3.2. Independence -Any dependencies of a tool that are not constrained in the way described in -*3.1* should be installed in a separate context, to avoid unnecessary -interference with the dependencies of the rest of the context. +Tool operations (`dune tools add`, `run`, etc.) must work correctly when a watch +server is running (`dune build -w`). Rather than directly manipulating lock +directories, tool commands should coordinate with the watch server via RPC to +avoid races and ensure the server picks up newly added tools. + +CR-soon Alizter: Specify the RPC protocol for tool operations. What messages are +needed? How does the watch server respond to tool additions? + +CR-soon Alizter: Document concurrent access behavior. What happens if two +terminals run `dune tools add` simultaneously? Or `add` while `run` is building? + +#### 3.5. Dog fooding + +Tools must work in the dune repository itself. Dune developers should be able to +run `dune tools add ocamlformat` and `dune tools add ocaml-lsp-server` when +working on dune, even though `dune pkg lock` doesn't work there due to the "in +and out" problem ([#8652]). + +[#8652]: https://github.com/ocaml/dune/issues/8652 + +This is enabled by the orthogonality design (3.2): tools are solved and built +independently from project dependencies, so they don't require a working project +lock directory. ### 4. UI #### 4.1. CLI -Users must be able to manage tools using the CLI interface. +Users must be able to manage tools using CLI commands: + +- Add/lock individual tools to the workspace +- Run tools (building if needed) +- List locked tools and versions +- Remove tools +- Discover paths to tool executables + +CLI-added tools persist until `dune clean` or `dune tools remove`, but are not +reproducible across clean builds. See [Batch operations](#batch-operations) for +batch commands. #### 4.2. Persistent configuration -When users wish to share a set of installed tools, or a particular tooling -configuration, they must be able to do so via configuration data in a file -(e.g., stanzas in a `dune-workspace` etc.). +Users must be able to declare tools in workspace configuration. Unlike CLI-added +tools, declared tools are reproducible: + +- Declaration survives `dune clean` (re-locked automatically from config) +- Declaration is version-controlled and shared with collaborators +- CI and fresh checkouts get the same tools + +See [The `(tool)` stanza](#the-tool-stanza) for syntax. + +### 5. Dune Integration + +Tools must integrate with existing dune features that rely on external +executables. + +#### 5.1. Format rules (`dune fmt`, `dune build @fmt`) + +`dune fmt` must use tools managed by `dune tools`. This includes respecting +version constraints from `.ocamlformat` files, PATH fallback +([2.4](#24-system-path-fallback)), and multi-version support +([1.4](#14-multi-version-support)). + +See [Tool resolution](#tool-resolution) for the resolution algorithm. + +#### 5.2. Documentation rules (`dune build @doc`, `dune ocaml doc`) + +`dune build @doc` must use tools managed by `dune tools`. When odoc is +configured as a tool, use the locked version; otherwise fall back to PATH +([2.4](#24-system-path-fallback)). + +See [Tool resolution](#tool-resolution) for the resolution algorithm. + +CR-soon Alizter: OCaml platform considerations + +#### 5.3. REPL (`dune utop`, `dune ocaml utop`) + +`dune utop` must use tools managed by `dune tools`. When utop is configured as +a tool, use the locked version; otherwise fall back to PATH +([2.4](#24-system-path-fallback)). + +See [Tool resolution](#tool-resolution) for the resolution algorithm. + +#### 5.4. Tool references in actions + +Build actions (both user-written rules and dune's internal rules like +formatting) must be able to reference tool executables via pforms. If the tool +is not locked, fall back to PATH (consistent with +[2.4](#24-system-path-fallback)). + +See [Tool pforms](#tool-pforms) in the Specification for syntax. + +#### 5.5. Legacy migration + +The legacy `.dev-tools.locks/` system is removed and replaced by this design. + +CR-soon Alizter: Detail the migration story: + +- Which CLI commands stay (with same or changed behavior)? +- Which CLI commands are removed? +- What happens to existing `.dev-tools.locks/` directories? +- User-facing migration guide (re-add tools via `dune tools add`) + +### 6. Non-requirements + +The following are explicitly out of scope: + +#### 6.1. System-wide installation + +Tools are workspace-local, not installed system-wide like `opam install` or +`cargo install`. There is no global tool registry outside of workspaces. + +Note: While tools are workspace-local, builds are still cached via the dune +build cache. If two workspaces install the same tool version with the same +compiler, the second workspace reuses the cached build artifacts. + +CR-someday Alizter: Consider system-wide tool installation for use outside +workspaces. Tools could be stored in `~/.cache/dune/tools/` and invoked via +`dune tools run` from any directory. This would enable: + +- Global utop without an opam switch +- Running OCaml scripts with dependencies (`dune run script.ml`) +- `dune shell` outside workspaces The same isolation and compiler-matching logic + applies. Related: [#12107] (system-wide packages). + +[#12107]: https://github.com/ocaml/dune/issues/12107 + +#### 6.2. Automatic updates + +Tools do not auto-update; explicit user action is required. + +#### 6.3. Tools for tools + +Using tools while developing the tools themselves (or their dependencies) has +nuances related to the "in and out" problem ([#8652]). + +**Different scenarios**: + +- **Working on a tool's dependency** (e.g., a library that ocamlformat uses): + You probably don't need ocamlformat to use your local changes - you just want + ocamlformat to work for formatting your code. The standard tool isolation + handles this fine. + +- **Working on the tool itself** (e.g., developing ocamllsp): You likely want to + use the ocamllsp you're building, not a separately installed version. This is + where pins could help - pin the tool to the local path. But circular builds + (tool needs to build itself) remain unsolved. + +The first scenario is common and supported. The second is out of scope until the +in-and-out problem is resolved, though pins may provide a partial workaround for +non-circular cases. + +#### 6.4. Environment manipulation + +No `dune tools env` command. Unlike `opam env`, we avoid stateful shell +environment manipulation. Use `dune tools run` or `dune tools path` instead. + +We acknowledge this adds friction for users migrating from opam and for +interactive shell use. However, we believe explicit invocation is preferable to +stateful environment that can silently become stale or inconsistent. + +CR-soon Alizter: There's a tension here: we reject `dune tools env` as too +implicit, but PATH fallback (2.4) is also implicit. Clarify the philosophy: is +the issue stateful shells specifically, or implicit behavior generally? + +CR-soon Alizter: dune tools env exists in the prototype, but it may be +undesirable. + +CR-someday Alizter: Consider `dune shell` as an alternative to `dune tools env`. +Instead of mutating the current shell's environment (opam-style), spawn a +subshell with tools in PATH (Nix-style). This avoids stale environment issues +and makes the boundary clear - just exit the subshell to return to normal. Could +also support `dune shell --with ocamlformat ocamllsp` syntax. + +#### 6.5. Precompiled binaries + +Tools are always built from source. Downloading precompiled binaries (like +cargo-binstall or uv's wheel downloads) is out of scope. + +This ensures compiler compatibility (Req 3.1) and avoids ABI (Application Binary +Interface) mismatch issues. OCaml binaries embed compiler-version-specific +formats - for example, `.cmi` files have checksums tied to compiler internals. A +pre-built tool (e.g., ocamllsp compiled with a different toolchain) cannot read +build artifacts from a differently-compiled project, causing "corrupted compiled +interface" errors ([#11229]). + +[#11229]: https://github.com/ocaml/dune/issues/11229 + +#### 6.6. Ephemeral runs + +No `uvx`-style temporary execution mode. Tools must be locked before running. + +Building OCaml tools from source is slow compared to downloading Python wheels, +so discarding the build after one use would be wasteful. Locking caches the +build for reuse. Users who want to try a tool temporarily can use +`dune tools add`, run it, then `dune tools remove`. + +CR-soon Alizter: Mention that autolocking makes this somewhat feasable, but it cannot +be relied on as it will roll to the newer versions. There are no binary +installs so therefore it would be slow also + +## Specification + +### The `(tool)` stanza + +_Implements: [1.3 Version pinning](#13-version-pinning), +[1.6 Binary selection](#16-binary-selection), +[3.1 Compiler compatibility](#31-compiler-compatibility), +[4.2 Persistent configuration](#42-persistent-configuration)_ + +Tools are declared in `dune-workspace` using the `(tool)` stanza. The stanza +uses the same dependency syntax as `(depends ...)` for the package field. + +#### Syntax + +```lisp +(tool + (package ) ; Required: package name with optional constraint + (executable ) ; Optional: binary name, defaults to package name + (skip_compiler_match) ; Optional: flag to disable compiler matching + (repositories ...)) ; Optional: restrict to specific repositories +``` + +CR-soon Alizter: missing pin + +CR-soon Alizter: Should share some of lock_dir stanza fields. Need to do a +comparison and explain why it is or is not a good idea. + +CR-soon Alizter: Edge cases for stanza parsing: + +- Duplicate stanzas for same package with different constraints: error, or + merge? +- Invalid package name syntax: error message? +- Stanza references non-existent repository: when to error (parse time vs solve + time)? + +#### Fields + +- **`(package )`** (required): The opam package providing the tool. Accepts + either a plain package name or a name with version constraint: + - Plain: `ocamlformat` + - With constraint: `(ocamlformat (= 0.26.2))` + - Version operators: `=`, `<>`, `<`, `>`, `<=`, `>=` + - Conjunctions: `(and (>= 0.25.0) (< 0.27.0))` + +- **`(executable )`** (optional): The binary to run. Defaults to the + package name. Required when the package provides multiple binaries. + +- **`(skip_compiler_match)`** (optional): When present, disables the default + compiler matching behavior. Use this for tools that don't need compiler + compatibility (e.g., formatters that only parse source text). + +CR-soon Alizter: The prototype uses `compiler_compatible` (opt-in) but this doc +says `skip_compiler_match` (opt-out). Reconcile naming and semantics. + +- **`(repositories )`** (optional): Restrict package resolution to the + named repositories. Useful for binary package repositories. + +CR-soon Alizter: The `(repositories)` field is parsed but verify that the +restriction is actually applied during solving in the prototype. + +CR-soon Alizter: Missing pins + +#### Compiler matching + +_Implements: [3.1 Compiler compatibility](#31-compiler-compatibility)_ + +Tools that read project build artifacts (e.g., ocamllsp reading `.cmi` files) +must be built with a compatible compiler to avoid ABI mismatches. By default, +tools are built with a compiler matching the environment. + +The compiler is detected in this order: + +1. **Project compiler** - if using dune package management with a `dune.lock/` +2. **System OCaml** - from opam switch or PATH +3. **Any compatible version** - if no compiler is detected + +When `(skip_compiler_match)` is set, the tool is solved without compiler +constraints, allowing it to use any OCaml version. + +CR-soon Alizter: Edge cases for compiler matching: + +- No compiler found (no opam switch, no system OCaml): error, or pick any? +- System OCaml version changes between `add` and `run`: rebuild, or warn? +- Tool requires newer OCaml than project: error message? + +#### Examples + +```lisp +;; Minimal: just the package name +(tool + (package ocamlformat)) + +;; With version constraint +(tool + (package + (ocamlformat + (= 0.26.2)))) + +;; With version range +(tool + (package + (ocamlformat + (and + (>= 0.25.0) + (< 0.27.0))))) + +;; Specifying the binary for multi-binary packages +(tool + (package menhir) + (executable menhirSdk)) + +;; Tool that doesn't need compiler matching +(tool + (package ocamlformat) + (skip_compiler_match)) + +;; Using a binary package repository +(tool + (package ocaml-lsp-server) + (repositories binary-packages)) +``` + +#### Invalid tool selections + +Some packages cannot be used as tools: + +- **Library-only packages**: Packages that provide no executables (e.g., `base`, + `lwt`) cannot be tools. Error at `dune tools add` time. + +CR-someday Alizter: The `%{tool:lib:...}` pform in [Tool pforms](#tool-pforms) +could enable library-only tool packages, but this needs further design work. + +CR-soon Alizter: How can we error at dune tools add time? We would only find out +after building the tool. + +- **Compiler packages**: Selecting `ocaml-base-compiler` or `ocaml-system` as a + tool is invalid. These conflict with the compiler matching logic. + +- **Dune itself**: Using `dune` as a tool is the "in and out" problem + ([non-requirement 6.3](#63-tools-for-tools)). Circular dependency issues make + this out of scope. + +CR-soon Alizter: The explanations for compiler packages and dune itself need +more thought. + +CR-soon Alizter: Specify exact error messages for these cases. + +### CLI commands + +_Implements: [1.3 Version pinning](#13-version-pinning), +[1.6 Binary selection](#16-binary-selection), +[2.1 Tool invocation](#21-tool-invocation), +[2.2 Programmatic use](#22-programmatic-use), [4.1 CLI](#41-cli)_ + +CR-soon Alizter: The CLI commands are generally underspecified. Each command +needs detailed behavior documentation: what triggers downloads/builds, how +errors are reported, interaction with stanzas vs CLI-added tools, exit codes, +output format, etc. + +``` +dune tools List all locked tools (alias for list) +dune tools add Lock all tools from (tool) stanzas +dune tools add [.] ... Lock specific tool(s) +dune tools run [.] [--bin ] [-- ] + Run a tool, passing arguments after -- +dune tools path [.] [--bin ] Print path to tool executable +dune tools list List all locked tools and versions +dune tools remove [.] ... Remove a tool's lock directory +``` + +The `--bin` flag is required when a package provides multiple binaries and no +`(executable ...)` is specified in the `(tool)` stanza. + +CR-soon Alizter: Edge cases for `dune tools add`: + +- Package doesn't exist in repository: error message? +- Network unavailable: error message, offline mode? +- Solver fails (unsatisfiable constraints): error with diagnostics? +- Build fails: partial state cleanup? +- Interrupted mid-install: cleanup, or resume on next run? +- Already locked at same version: no-op, or re-solve? +- Already locked at different version: add second version, or replace? + +CR-soon Alizter: The prototype has `--allow-not-installed` flag for +`dune tools path`. Document this flag and its use case (editor integration). + +#### Version syntax + +The CLI uses dot-separated version syntax: `.` + +```bash +dune tools add ocamlformat # Latest version +dune tools add ocamlformat.0.26.2 # Specific version +``` + +#### Batch operations + +Batch operations apply to commands that manage tool state, not to commands that +use tools: + +| Command | Batch | Rationale | +| ------------------- | ----- | ----------------------------------------------- | +| `dune tools add` | ✓ | No args: all from stanzas; with args: specified | +| `dune tools remove` | ✓ | Multiple packages | +| `dune tools run` | ✗ | Running is inherently single-target | +| `dune tools path` | ✗ | Path lookup is per-tool | + +CR-soon Alizter: Edge cases for `dune tools remove`: + +- Tool not locked: error, or no-op with warning? +- Tool declared in stanza: remove lock but warn it will be re-added? +- Multiple versions locked: remove all, or require version specifier? +- Remove while tool is running: behavior? + +**`dune tools add`** has two modes: + +```bash +dune tools add # Lock all from (tool) stanzas +dune tools add ocamlformat odoc # Lock specific packages +dune tools add ocamlformat.0.26.2 odoc.2.4.0 # With version constraints +``` + +The no-argument form is critical for CI and onboarding - a single command to set +up all project tooling from declarative configuration. + +CR-someday Alizter: Integration with opam's `:with-dev-setup` marker ([#12135]): + +[#12135]: https://github.com/ocaml/dune/issues/12135 + +Projects often declare their development tools in opam files using the +`:with-dev-setup` filter (e.g., `ocamlformat :with-dev-setup`). Currently, dune +tools doesn't read these - users must manually add each tool or duplicate the +declarations in `(tool)` stanzas. + +One possible approach: + +```bash +dune tools add --with-dev-setup # Install tools from opam :with-dev-setup deps +dune tools add --with-doc # Install tools from :with-doc deps +``` + +This would enable: + +- Single source of truth for project tooling (opam file) +- Automatic tool discovery for new contributors +- CI scripts that don't need to know which tools to install + +CR-soon Alizter: Is `--with-dev-setup` the right approach? Alternatives: + +- Automatic: `dune tools add` (no args) always includes `:with-dev-setup` deps +- Separate command: `dune tools setup` specifically for opam marker integration +- No integration: keep `(tool)` stanzas as the only source, avoid duplication by + not using `:with-dev-setup` in opam files at all The flag approach adds + complexity. Need to evaluate if the use case justifies it. + +CR-soon Alizter: If we do integrate, design questions: + +- Read from `dune-project` `(depends)` or from `.opam` files? +- What if opam file says `ocamlformat` but `.ocamlformat` says `version=0.26.2`? +- How does this interact with `(tool)` stanzas that specify the same package? + +CR-soon Alizter: The prototype requires at least one package argument. The +no-argument batch mode (lock all from stanzas) is not yet implemented. + +#### Open question: version separator + +The current prototype uses `.` as the version separator (`pkg.version`), aligned +with opam conventions. However, `@` is common in other ecosystems: + +- **npm/npx**: `npx cowsay@1.5.0` +- **cargo**: `cargo install ripgrep@14.0.0` + +Opam package names cannot contain dots (allowed: `[a-zA-Z0-9_+-]+`), so `.` is +unambiguous. The choice is primarily about familiarity vs ecosystem consistency. + +#### Open question: `add` vs `install` naming + +The current prototype uses `dune tools add` for adding individual tools. +However, the naming convention is still open for discussion: + +- **Option A**: `add` for individual, `install` for batch (current prototype) +- **Option B**: `install` for both (consistent with legacy, npm, cargo) +- **Option C**: `lock` to emphasize the solving/locking aspect +- **Option D**: Semantic continuum where commands differ by how much work they + do: + - `add` = lock only (solve + write lock) + - `install` = lock + build (download + compile) + - `run` = lock + build + execute This gives users control over when work + happens (useful for CI, Docker builds). + +Considerations: + +- Legacy issues reference `dune tools install` +- `add` suggests incremental, `install` suggests setup +- `lock` aligns with `dune pkg lock` but may confuse (tools also build) +- Option D aligns with user expectations: "add to config" vs "install software" + +#### Open question: `dune tools run` PATH fallback + +Should `dune tools run ` fall back to system PATH when the tool is not +locked? + +- **Option A**: Fall back to PATH (consistent with build rules like `dune fmt`) +- **Option B**: Error with suggestion to lock or install on PATH (current impl) + +Considerations: + +- Falling back makes `dune tools run` a universal "run this tool" command +- Erroring is more explicit and avoids silently using unexpected versions +- Build rules need PATH fallback for gradual adoption (e.g., `dune fmt` without + locking ocamlformat) +- CLI commands could be stricter since the user explicitly requested a tool + +### Tool resolution + +_Implements: [1.4 Multi-version support](#14-multi-version-support), +[2.4 System PATH fallback](#24-system-path-fallback), +[5.1 Format rules](#51-format-rules-dune-fmt)_ + +Tool resolution answers two questions: + +1. **Source**: Where does the tool come from? +2. **Version**: Which version of the tool? + +#### Source resolution + +When dune needs a tool, it checks these sources in order: + +1. **Workspace `(tool)` stanza** - declared in `dune-workspace` +2. **Locked versions** - previously added via `dune tools add` +3. **System PATH** - fallback if nothing else found + +The first source that has the tool wins. If no source has it, dune falls back to +system PATH (which may result in "command not found"). + +#### Version selection + +CR-soon Alizter: Version conflict resolution algorithm is undefined. What +happens when multiple sources specify conflicting versions? E.g., stanza says +`(= 0.26.2)` but `.ocamlformat` says `version=0.27.0`. Need to define precedence +or error behavior. + +How the version is determined depends on context: + +**CLI commands** (`dune tools run`, `dune tools add`): + +- Explicit version: `dune tools run ocamlformat.0.26.2` uses that version +- No version: uses the single locked version, or errors if multiple exist + +**`(tool)` stanza**: + +- Version constraint in stanza: `(package (ocamlformat (= 0.26.2)))` +- Constraint applied when locking, not at runtime + +**Formatting rules** (`dune fmt`): + +1. Read `version` field from `.ocamlformat` file +2. If version present: use `%{tool:ocamlformat.}` pform +3. If no version: fall back to `%{bin:ocamlformat}` (system PATH) + +The pform handles locking/building/PATH fallback per the resolution rules above. + +#### Example scenarios + +| Scenario | Source | Version | +| ------------------------------------------- | -------------- | -------------- | +| `(tool (package ocamlformat))` in workspace | stanza | locked version | +| `dune tools add ocamlformat.0.26.2` | locked | 0.26.2 | +| `.ocamlformat` says `version=0.26.2` | locked or PATH | 0.26.2 | +| No config, ocamlformat on PATH | PATH | system version | + +CR-soon Alizter: Specify error messages for failure modes: solver fails, network +error, build fails, binary not found, multiple binaries without --bin, etc. Is +the locked version the most recent? + +### Directory structure + +_Implements: [1.2 Workspace-local](#12-workspace-local), +[1.5 Clean source tree](#15-clean-source-tree), +[3.2 Dependency isolation](#32-dependency-isolation)_ + +Tools use three types of directories: + +1. **External lock directories** - Created by `dune tools add`, contain solver + output +2. **Build lock directories** - Internal copies for build rules +3. **Install directories** - Built artifacts (binaries, libraries) + +``` +_build/ +├── .tools.lock/ # External lock directories (solver output) +│ └── / +│ └── / +│ ├── lock.dune # Lock file +│ └── .pkg # Package metadata +└── _private/ + └── default/ + ├── .tool-locks/ # Build lock directories (internal) + │ └── / + │ └── / + │ ├── lock.dune + │ └── .pkg + └── .tools/ # Install directories (built artifacts) + └── / + └── / + └── target/ + ├── bin/ # Installed executables + └── cookie # Installation trace (artifacts + variables) +``` + +**Path flow**: External lock → copied to build lock → built to install dir. + +**Key paths**: + +- External lock: `_build/.tools.lock///` +- Build lock: `_build/_private/default/.tool-locks///` +- Install: `_build/_private/default/.tools///target/` + +Note: The external lock directory (`_build/.tools.lock/`) is at the build root, +not inside `_private/`. This allows tools to be locked without going through the +build system. The build lock directory and install directory are both inside +`_private/default/` as they are managed by the build system. + +### Tool pforms + +_Implements: [5.4 Tool references in actions](#54-tool-references-in-actions)_ + +Build actions can reference tool executables via `%{tool:...}` pforms. + +#### Syntax + +``` +%{tool:} ; executable (default binary) +%{tool:.} ; specific version +``` + +#### Resolution + +1. If tool is locked, use the locked executable path +2. If tool is not locked, fall back to PATH (like `%{bin:...}`) + +CR-soon Alizter: What happens when version is omitted and multiple versions are +locked? Error, or pick one? + +#### Examples + +```lisp +;; Run a tool executable +(rule + (action (run %{tool:ocamlformat} %{deps}))) + +;; Use a specific version +(rule + (action (run %{tool:ocamlformat.0.26.2} %{deps}))) +``` + +#### Future: library access + +A future extension could allow accessing libraries from tool packages via +`%{tool:lib:}`. This would enable using a library solved independently from +the project's dependency graph. + +However, this has significant drawbacks: if two tool libraries share a common +dependency (e.g., both depend on `base`), they are solved independently and may +have incompatible versions. Using both in the same build rule could cause +linking errors. This needs further design work around "tool groups" or coherence +constraints before being viable. + +## Relationship to Package Management + +Tools are built on the same infrastructure as `dune pkg` (Dune's package +management): + +- Same solver (`opam-0install-solver`) +- Same repository system (opam-repository, overlays, custom repos) +- Same lock file format +- Same build rules for fetching and compiling packages + +### Orthogonality: a key design principle + +Tools are **orthogonal** to project package management. This decoupling +simplifies the design: + +- **No project lock required**: Tools work even without `dune.lock/`. A project + can use `dune tools` without enabling package management. +- **Independent solving**: Adding/removing tools doesn't affect project + dependencies, and vice versa. +- **No build invalidation**: Tool changes don't trigger project rebuilds + ([requirement 3.3](#33-no-build-triggers)). +- **Multi-version is natural**: Since each tool version is solved independently, + supporting multiple versions falls out of the design. + +The legacy system violated this principle by mixing tool dependencies with +project packages (the "wrapper package hack"), causing solver conflicts and +complex regeneration logic. + +### The package-tool continuum + +Rather than a binary choice between "package" and "tool", the design supports a +continuum of integration levels: + +**Fully integrated (packages)**: Dependencies in `dune.lock/` are solved +together, libraries and executables are available throughout the build, and +changes affect the entire project. + +**Fully isolated (tools)**: Each tool has its own lock directory, is solved +independently, and by default contributes nothing to the build environment. The +tool sits in `_build/` invisible to the project unless explicitly invoked. + +**Selective exposure (via pforms)**: The `%{tool:...}` pform +([Tool pforms](#tool-pforms)) allows build rules to selectively reference tool +executables. This lets a tool remain isolated (independently solved, no +dependency conflicts) while still being usable in specific rules. + +This continuum lets users choose the right level of integration: use packages +when you need tight coupling and shared dependencies; use tools when you want +isolation and version flexibility; use pforms when you need something in +between. + +### Key differences from project dependencies + +| Aspect | Project packages (`dune pkg`) | Tools (`dune tools`) | +| ------------- | --------------------------------- | ---------------------------------- | +| Lock location | `dune.lock/` (committed) | `_build/.tools.lock/` (ephemeral) | +| Solving | All packages solved together | Each tool solved independently | +| User workflow | `dune pkg lock` then `dune build` | `dune tools add`, builds on demand | +| Compiler | Project's compiler | Matches project or system OCaml | +| Requires lock | Yes | No (can use system OCaml) | + +CR-soon Alizter: system-ocaml, the opam package, is a bit of a hack. If a user +changes their opam switch it might not work as intended. + +CR-soon what does the Requires lock even mean here? + +### Why tools don't use `dune.lock/` + +Tools are stored in `_build/` rather than alongside `dune.lock/` because: + +1. **Tools are workspace infrastructure**, not project deliverables +2. **Different sub-projects may need different tool versions** (e.g., different + `.ocamlformat` versions per directory) +3. **Avoids cluttering source tree** (addresses [#10955], [#12097]) + +[#10955]: https://github.com/ocaml/dune/issues/10955 +[#12097]: https://github.com/ocaml/dune/issues/12097 + +The tradeoff is that tool lock directories are not committed to version control. +Reproducibility comes from `(tool)` stanzas in `dune-workspace`, which declare +version constraints that are re-solved when `_build/` is deleted. + +### Locking vs building + +These are separate phases, though `dune tools add` combines them: + +1. **Locking**: Solve dependencies, write lock directory with package metadata +2. **Building**: Download sources, compile tool, install to target directory + +Building (which includes downloading sources) happens lazily on first +`dune tools run`, not during `dune tools add`. The `add` command only solves +dependencies and writes the lock directory. + +**Open question**: How can users download, build, and install tools without +running them? This is useful for CI (prepare all tools upfront) or offline +scenarios. Options include a `--build` flag on `add`, a separate +`dune tools build` command, or relying on `dune tools path` to trigger builds. + +### Future: Relocatable compiler and caching + +Currently, each workspace builds its own copy of tools, even if the same tool +version is used across multiple projects. This is wasteful. + +The **relocatable compiler** work will enable: + +- **Shared build cache**: Tool builds can be cached and reused across projects +- **Faster tool installation**: If another project already built + `ocamlformat.0.26.2` with the same compiler, reuse the cached artifacts +- **Binary distribution**: Pre-built tool binaries could be distributed via + binary package repositories + +This will address the performance concerns of +[non-requirement 6.1](#61-system-wide-installation) (System-wide installation) +while maintaining project isolation. + +### Potential future stanza options + +The `(lock_dir)` stanza for project packages supports options not yet available +for tools: + +- `(solver_env ...)` - inject variables into solver +- `(constraints ...)` - additional solver constraints +- `(version_preference newest|oldest)` - prefer newest or oldest versions +- `(pins ...)` - pin packages to specific sources + +These may be added to `(tool)` stanzas if use cases emerge. In particular, +`(pins ...)` would help with [non-requirement 6.3](#63-tools-for-tools) (Tools +for tools) by allowing tools to be pinned to local paths. + +## GitHub Issues Addressed + +This section maps known issues to the requirements that address them. + +### Arbitrary tool packages ([#12913]) + +**Problem**: The current design requires every tool to be hardcoded. Only a few +tools are supported, and adding new ones requires code changes to dune. + +**Addressed by**: Requirement 1.1 (Generality). Any opam package that provides +an executable can be installed as a tool via `dune tools add `. + +[#12913]: https://github.com/ocaml/dune/issues/12913 + +### Lock directory clutter ([#10955], [#12097]) + +**Problem**: Users complained about lock directories cluttering the source tree. + +**Addressed by**: Requirement 1.5 (Clean source tree). Tool lock directories are +now stored in `_build/.tools.lock/`, not the source tree. + +[#10955]: https://github.com/ocaml/dune/issues/10955 +[#12097]: https://github.com/ocaml/dune/issues/12097 + +### Compiler ABI mismatch ([#11229]) + +**Problem**: Pre-built tool binaries (musl toolchain) don't work with +locally-built OCaml compilers because interface hashes don't match. + +**Addressed by**: Requirement 3.1 (Compiler compatibility). Tools are built from +source with matching compiler constraints by default. + +[#11229]: https://github.com/ocaml/dune/issues/11229 + +### Graceful fallback for missing tools ([#10578]) + +**Problem**: `dune fmt` fails hard when ocamlformat isn't installed, even if +other formatters could run. + +**Addressed by**: Requirements 2.4 (System PATH fallback) and 5.1 (Format +rules). When no version is specified in `.ocamlformat`, dune falls back to PATH. +If not found, the OCaml formatting rules are skipped rather than failing the +entire build. + +[#10578]: https://github.com/ocaml/dune/issues/10578 + +### Version constraints ([#12777]) + +**Problem**: No robust way to specify version constraints for dev tools. Users +had to hardcode paths to lock directories. + +**Addressed by**: Requirements 1.3 (Version pinning) and 4.2 (Persistent +configuration). The `(tool)` stanza allows declarative version constraints: + +```lisp +(tool + (package + (ocamlformat + (= 0.26.2)))) +``` + +[#12777]: https://github.com/ocaml/dune/issues/12777 + +### Version consistency ([#10688]) + +**Problem**: When ocamlformat isn't a project dependency, `dune fmt` captures +whatever version is on PATH, leading to inconsistent formatting. + +**Addressed by**: Requirements 1.3 (Version pinning) and 5.1 (Format rules). +When `.ocamlformat` specifies a version, dune uses the locked version. The PATH +is only used as fallback when no version is specified. + +[#10688]: https://github.com/ocaml/dune/issues/10688 + +### Dev tools redesign ([#12914]) + +**Problem**: Meta-issue tracking the need for a complete redesign of dev tools. + +**Addressed by**: This entire design document. Key improvements: + +- Generality (1.1): Any package, not just hardcoded ones +- Clean source tree (1.5): Lock dirs in `_build/` +- Version pinning (1.3): Declarative constraints via stanzas +- Multi-version (1.4): Different projects can use different versions +- Compiler compatibility (3.1): Tools built with matching compiler + +[#12914]: https://github.com/ocaml/dune/issues/12914 + +### Tool dependency isolation ([#12551]) + +**Problem**: Tools and project packages conflict when they share dependencies. + +**Addressed by**: Requirements 3.2 (Dependency isolation) and 5.3 (REPL). Each +tool has its own lock directory and is solved independently from project +dependencies. + +[#12551]: https://github.com/ocaml/dune/issues/12551 + +### No project build triggers ([#11037]) + +**Problem**: Running `dune fmt` triggers a full build of project dependencies, +even though formatting only needs ocamlformat. + +**Addressed by**: Requirements 3.3 (No build triggers) and 5.1 (Format rules). +Tool operations are isolated from project builds. + +[#11037]: https://github.com/ocaml/dune/issues/11037 + +### Compiler-independent formatters ([#11038]) + +**Problem**: `dune fmt` fails if no OCaml compiler is available, even though +ocamlformat could be a standalone binary. + +**Addressed by**: Requirement 3.1 (Compiler compatibility). Tools match the +project or system compiler, but formatters can opt out via +`(skip_compiler_match)`. + +[#11038]: https://github.com/ocaml/dune/issues/11038 + +### Version from config files ([#5315]) + +**Problem**: The `version` field in `.ocamlformat` should determine which +ocamlformat version to use. Different projects within a workspace may have +different `.ocamlformat` files specifying different versions. + +**Addressed by**: Requirements 1.3 (Version pinning), 1.4 (Multi-version +support), and 5.1 (Format rules). Formatting rules resolve the correct +ocamlformat version based on each directory's `.ocamlformat` file, and multiple +versions can coexist in the workspace. + +[#5315]: https://github.com/ocaml/dune/issues/5315 + +### Version constraint parsing ([#12866]) + +**Problem**: Version constraints for dev tools don't work correctly. + +**Addressed by**: Requirement 1.3 (Version pinning). The new `(tool)` stanza +properly supports version constraints using dependency syntax. + +[#12866]: https://github.com/ocaml/dune/issues/12866 + +### Opam with-dev-setup integration ([#12135]) + +**Problem**: Projects declare development tools in opam files using the +`:with-dev-setup` filter, but dune tools doesn't integrate with this. Users must +manually discover and add each tool, or duplicate declarations in both opam +files and `(tool)` stanzas. + +**Addressed by**: Requirement 4.1 (CLI), future work. See "Batch operations" in +the Specification for the planned `--with-dev-setup` flag. + +[#12135]: https://github.com/ocaml/dune/issues/12135 + +### Helpful error messages ([#13235], [#12975]) + +**Problem**: When tools aren't installed, error messages don't guide users +toward the correct commands. + +**Addressed by**: Requirement 4.1 (CLI). Commands should provide helpful +suggestions when tools aren't installed. + +[#13235]: https://github.com/ocaml/dune/issues/13235 +[#12975]: https://github.com/ocaml/dune/issues/12975 + +### Batch installation ([#12557]) + +**Problem**: Installing multiple tools requires multiple commands. + +**Addressed by**: Requirement 4.1 (CLI). `dune tools add` accepts multiple +package arguments, and with no arguments locks all tools from `(tool)` stanzas. +See "Batch operations" in the Specification. + +[#12557]: https://github.com/ocaml/dune/issues/12557 + +### Failure isolation ([#12818]) + +**Problem**: Failed tool installation can leave the environment in a broken +state. + +**Addressed by**: Requirement 3.2 (Dependency isolation). Each tool is isolated, +so a failed install doesn't affect other tools or the project. + +[#12818]: https://github.com/ocaml/dune/issues/12818 + +### Utop findlib integration ([#13471]) + +**Problem**: Running `#require` in utop triggers findlib warnings about +non-existent directories from dune's sandboxed package paths. + +**Addressed by**: Requirement 5.3 (REPL). When utop is configured as a tool, it +should be properly integrated with dune's package environment. + +[#13471]: https://github.com/ocaml/dune/issues/13471 + +## Comparison with Other Tools + +This section documents how other ecosystems handle tool management, informing +our design decisions and identifying gaps in our approach. + +### uv (Python) + +[uv][uv docs] distinguishes between ephemeral and persistent tool usage: + +- **`uvx `**: Runs tool in temporary virtual environment, cached but + disposable +- **`uv tool install `**: Persistent installation, executables on PATH + +**Key features**: + +- Per-tool isolated virtual environments (no cross-tool conflicts) +- Version syntax: `uvx ruff==0.3.0` (pinned), `uvx ruff@latest` (latest only) +- Constraint preservation: `uv tool install black>=23,<24` is respected by + `uv tool upgrade` + +**Relevance to requirements**: + +- **3.2 Dependency isolation**: Per-tool venvs achieve isolation. Our separate + lock dirs provide similar isolation. + +**Gap**: No project-local tool declaration. Tools are always user-global. This +violates our [requirement 1.2](#12-workspace-local) (Workspace-local). + +**What we should adopt**: + +- `upgrade` command that respects original constraints +- Ephemeral vs persistent distinction + +CR-soon Alizter: `dune tools upgrade` is not in the CLI specification. Add it or +note as future work. + +### cargo (Rust) + +[cargo install][cargo docs] provides minimal built-in tool support: + +- All binaries installed to single `~/.cargo/bin/` directory +- No isolation between tools +- No multi-version support (new version overwrites old) +- No project-local tools + +**Key features**: + +- Version constraint syntax: `cargo install ripgrep@1.2.0`, `--version ~1.2` +- Smart reinstall: only rebuilds if version/features/profile changed +- `cargo install --list` shows all installed packages + +**Relevance to requirements**: + +- **1.2 Workspace-local**: Violated. Cargo's model is entirely system-wide. +- **3.2 Dependency isolation**: Violated. All tools share one bin directory. + +**Gap**: No project-local tools, which is why cargo-run-bin exists. + +### cargo-run-bin (Rust community tool) + +[cargo-run-bin] is the **closest analog to our design**. It fills cargo's gaps +with project-local tool management: + +```toml +[package.metadata.bin] +cargo-nextest = { version = "0.9.57", locked = true } +dprint = { version = "0.30.3" } +cargo-mobile2 = { version = "0.5.2", bins = ["cargo-android", "cargo-mobile"] } +``` + +**Key features**: + +- Project-local `.bin/` cache directory +- Declarative config in `Cargo.toml` metadata section +- `bins` array for multi-binary packages +- `locked` flag for reproducible dependency resolution +- `cargo bin --install` to install all configured tools at once + +**Relevance to requirements**: + +- **1.2 Workspace-local**: ✓ Achieves this via `.bin/` directory +- **4.2 Persistent configuration**: ✓ Uses `Cargo.toml` metadata + +**What we should adopt**: + +- `bins` field for explicit binary listing (vs auto-discovery) +- `--locked` flag for reproducibility +- **Batch install command** (`dune tools add`). Critical for CI + +### npm (JavaScript) + +[npm][npm docs] has the most mature approach for project-local tools: + +- Tools declared in `devDependencies` in `package.json` +- Binaries linked to `node_modules/.bin/` +- `npx` runs binaries, preferring local over global +- npm scripts automatically get `node_modules/.bin` in PATH +- `package-lock.json` provides exact version pinning + +**Relevance to requirements**: + +- **1.2 Workspace-local**: ✓ `node_modules/.bin/` is project-local +- **2.2 Programmatic use**: ✓ Single directory for all tool binaries +- **4.2 Persistent configuration**: ✓ `devDependencies` in package.json + +**What we should adopt**: + +- devDependencies pattern (tools as dev deps, committed to repo) +- Auto-PATH for scripts (tools available without explicit path) +- `npx` prefers local, falls back to fetch + +### How Our Design Compares + +**Note**: Dune entries below describe **design requirements**, not necessarily +fully implemented features. Each entry references the relevant requirement +(e.g., "Req 1.2") to enable verification against the specification. + +#### Tools compared + +**uv** (Python): Fast Python package manager with tool support. Tools are +installed globally with per-tool isolated virtual environments. Supports +ephemeral runs (`uvx`) and constraint-preserving upgrades. No project-local or +declarative configuration. ([uv docs]) + +**cargo** (Rust): Rust's built-in package manager. Minimal tool support. All +binaries go to `~/.cargo/bin/`, no isolation, no multi-version. Smart reinstall +detects version changes. Compiler matching via rustup's `rust-toolchain.toml`. +([cargo docs]) + +**cargo-run-bin** (Rust): Community tool filling cargo's gaps. Project-local +`.bin/` cache, declarative config in `Cargo.toml` metadata, batch install. +Closest analog to our design in the Rust ecosystem. ([cargo-run-bin]) + +**npm** (JavaScript): Most mature project-local approach. Tools in +`devDependencies`, binaries in `node_modules/.bin/`, `npx` for execution. +Semver-respecting upgrades, lock file for reproducibility. ([npm docs]) + +#### Comparison table + +| Feature | dune | uv | cargo | cargo-run-bin | npm | +| -------------------- | ---- | --- | ----- | ------------- | --- | +| Project-local | ✓ | ✗ | ✗ | ✓ | ✓ | +| Multi-version | ✓ | ✗ | ✗ | ✗ | ✓\* | +| Declarative config | ✓ | ✗ | ✗ | ✓ | ✓ | +| Compiler matching | ✓ | ✗ | ✓\* | ✗ | ✗ | +| PATH fallback | ✓ | ✗ | ✗ | ✗ | ✓ | +| Precompiled binaries | ✗ | ✓ | ✗ | ✓\* | ✓ | +| Tool isolation | ✓ | ✓ | ✗ | ✓ | ✓ | +| Ephemeral runs | ✗ | ✓ | ✗ | ✗ | ✓ | +| Batch install | ✓ | ✗ | ✗ | ✓ | ✓ | +| Binary selection | ✓ | ✓ | ✓ | ✓ | ✓ | +| Tool discovery | ✓ | ✓ | ✗ | ✗ | ✓ | +| Tool upgrade | ✗ | ✓ | ✓\* | ✗ | ✓ | +| System-wide install | ✗ | ✓ | ✓ | ✗ | ✓ | + +_Asterisks indicate caveats; see relevant subsection for details._ + +#### Project-local + +Tools are installed within the project directory, isolated from other projects +and the global system. + +**Dune** (Req 1.2, 1.5): Tools are workspace-local. Lock files stored in +`_build/.tools.lock///`, built artifacts in +`_build/_private/default/.tools///`. + +**uv**: ✗ Tools installed globally. "Each tool environment is linked to a +specific Python version." Multiple Python versions require separate installs +with `--python` flag. If the Python version is uninstalled, the tool breaks. +([uv docs]) + +**cargo**: ✗ All binaries installed to single `~/.cargo/bin/`, shared by all +projects. Only one version per tool; reinstalling overwrites. Different versions +require custom `CARGO_INSTALL_ROOT` per project. ([cargo docs]) + +**cargo-run-bin**: ✓ Project-local `.bin/` cache directory. "Designed to manage +your Rust binaries within the context of your project." ([cargo-run-bin]) + +**npm**: ✓ Binaries linked to `node_modules/.bin/` per project. "Executables +linked into `./node_modules/.bin`." ([npm docs]) + +#### Multi-version + +Multiple versions of the same tool can coexist and be used within a single +workspace. + +**Dune** (Req 1.4): Versioned paths `_build/.tools.lock///` allow +multiple versions. Version selection: if 1 version locked, use it; if multiple, +error unless specified (e.g., `.ocamlformat` version field selects for +formatting). + +**uv**: ✗ "Installing a tool with the same name will replace the existing tool." +Open feature request for per-directory pinning (GitHub #17813). ([uv docs]) + +**cargo**: ✗ "If the package is already installed, Cargo will reinstall it if +the installed version does not appear to be up-to-date." Only one version per +install root; use `--force` to explicitly switch versions. ([cargo docs]) + +**cargo-run-bin**: ✗ Single version per tool in `Cargo.toml` metadata. +([cargo-run-bin]) + +**npm**: ✓\* Nested `node_modules/` can contain different versions for +transitive dependencies. Package aliases allow explicit multi-version: +`npm install react15@npm:react@^15`. For CLI tools, `npx pkg@version` runs +specific versions. ([npm docs: folders]) + +#### Declarative config + +Tools can be declared in a configuration file, enabling reproducible setups. + +**Dune** (Req 4.2): `(tool)` stanza in `dune-workspace`: + +```lisp +(tool + (package + (ocamlformat + (= 0.26.2)))) +``` + +**uv**: ✗ No project-level tool configuration. Tools installed via CLI only. +([uv docs]) + +**cargo**: ✗ No built-in declarative tool configuration. Feature request was +explicitly rejected as "NOT_PLANNED" - Cargo team directed users to third-party +tools instead. ([cargo#5120]) + +**cargo-run-bin**: ✓ Declared in `Cargo.toml` metadata section: + +```toml +[package.metadata.bin] +cargo-nextest = { version = "0.9.57" } +``` + +([cargo-run-bin]) + +**npm**: ✓ Declared in `package.json` devDependencies: + +```json +{ "devDependencies": { "prettier": "^3.0.0" } } +``` + +([npm docs: package.json]) + +#### Compiler matching + +Tools are built with a compiler version matching the project to ensure ABI +compatibility. + +**Dune** (Req 3.1): Automatic by default. Tools match project compiler or system +OCaml. Opt-out via `(skip_compiler_match)`. + +**uv**: ✗ Tools ignore project-level Python version settings. "Tool environment +details: Each tool environment is linked to a specific Python version... but +will ignore non-global Python version requests like `.python-version` files and +the `requires-python` value from a `pyproject.toml`." ([uv docs]) + +**cargo**: ✓* Via rustup (not cargo itself): `rust-toolchain.toml` in the +*current directory* affects `cargo install`. Note: this uses *your project's\* +toolchain, not the tool's - a known issue ([rust-lang/cargo#11036]). ([rustup +docs]) + +[rust-lang/cargo#11036]: https://github.com/rust-lang/cargo/issues/11036 + +**cargo-run-bin**: ✗ Uses whatever Rust toolchain is active. + +**npm**: ✗ No automatic matching. Packages declare Node version via `engines` +field, but npm only warns by default (requires `.npmrc` `engine-strict=true` to +enforce). Native addons use ABI version matching via node-pre-gyp. ([npm docs: +package.json]) + +#### PATH fallback + +When a tool is not installed via the tool manager, fall back to system PATH. + +**Dune** (Req 2.4): Build rules (e.g., `dune fmt`) fall back to system PATH when +no tool is locked. Behavior for `dune tools run` is an open question (see +below). + +**uv**: ✗ Does not search system PATH for tools. + +**cargo**: ✗ `cargo install` manages `~/.cargo/bin/`, but individual commands +don't fall back to system PATH. + +**cargo-run-bin**: ✗ Only runs binaries from local `.bin/` cache. + +**npm**: ✓ npx checks `$PATH` and `node_modules/.bin` before downloading. Use +`--ignore-existing` to skip PATH lookup. (Note: "shell fallback" was a different +feature—`--shell-auto-fallback`—removed in npm v7.) ([npm v7 changes]) + +#### Precompiled binaries + +Tools can be downloaded as precompiled binaries rather than built from source. + +**Dune** (Non-req 6.5): ✗ Tools always built from source. Ensures compiler +compatibility and avoids ABI issues. + +**uv**: ✓ Downloads prebuilt wheels when available. "uv downloads and installs +packages from PyPI." ([uv docs]) + +**cargo**: ✗ `cargo install` compiles from source. ([cargo docs]) + +**cargo-run-bin**: ✓\* Optional binstall support. "Download pre-compiled +binaries using cargo-binstall." ([cargo-run-bin]) + +**npm**: ✓ Packages are prebuilt JavaScript. Native addons may require +compilation. + +#### Tool isolation + +Each tool's dependencies are isolated from other tools and from project +dependencies. + +**Dune** (Req 3.2): Each tool solved independently with its own lock directory +in `_build/.tools.lock///`. + +**uv**: ✓ "Each tool has its own isolated virtual environment." ([uv docs]) + +**cargo**: ✗ All tools share single `~/.cargo/bin/` directory. No dependency +isolation between tools. + +**cargo-run-bin**: ✓ Each tool installed to separate directory under `.bin/`. +([cargo-run-bin]) + +**npm**: ✓ Each project has isolated `node_modules/`. Tools share dependencies +within a project but are isolated between projects. + +#### Ephemeral runs + +Run a tool once without persisting its installation - useful for trying tools or +running one-off commands without polluting the workspace's tool configuration. + +**Dune** (Non-req 6.6): ✗ Tools must be locked before running. Building from +source is slow, so caching (locking) is more practical than discarding. Users +can `dune tools remove` after if needed. + +**uv**: ✓ `uvx` runs in temporary environment. "Runs a tool without installing +it... The tool environment is cached." ([uv docs: uvx]) + +**cargo**: ✗ No ephemeral execution; `cargo install` always persists. + +**cargo-run-bin**: ✗ Tools must be configured and installed first. + +**npm**: ✓ `npx` can fetch and run without adding to package.json. "Run a +command from a local or remote npm package." ([npx docs]) + +#### Batch install + +Install all declared tools with a single command. + +**Dune** (Req 4.1): `dune tools add` with no arguments locks all tools from +`(tool)` stanzas. + +**uv**: ✗ No batch install; tools installed individually via `uv tool install`. + +**cargo**: ✗ No declarative tool configuration, so no batch install. + +**cargo-run-bin**: ✓ `cargo bin --install` installs all tools from config. +"cargo bin --install - Installs all binaries." ([cargo-run-bin]) + +**npm**: ✓ `npm install` installs all devDependencies. "Install all modules +listed as dependencies in package.json." ([npm docs]) + +#### Binary selection + +When a package provides multiple executables, how to specify which one to run. + +**Dune** (Req 1.6): `--bin` flag on `dune tools run` and `dune tools path`. +Auto-selects if package provides single binary; errors with helpful message if +multiple binaries exist. Declarative via `(executable ...)` in `(tool)` stanza. + +**uv**: ✓ `--from` option specifies package when command differs from package +name: `uvx --from httpie http`. Without `--from`, uvx assumes command name +equals package name. `uv tool install` installs all binaries from a package. +([uv docs]) + +**cargo**: ✓ `--bin NAME` flag selects specific binary. Installs all binaries by +default. `default-run` in Cargo.toml only affects `cargo run`, not install. +Binary name conflicts between packages cause errors. ([cargo docs]) + +**cargo-run-bin**: ✓ `bins` array in config specifies which binaries to build: +`cargo-mobile2 = { version = "0.5.2", bins = ["cargo-android", "cargo-mobile"] }`. +Builds all binaries if `bins` omitted. ([cargo-run-bin]) + +**npm**: ✓ `--package` (or `-p`) specifies package when binary name differs: +`npx --package=foo bar`. npx auto-matches binary to package name; errors if no +match. `bin` field in package.json maps command names to scripts. ([npx docs]) + +#### Tool discovery + +How scripts and editors programmatically find paths to installed tool +executables. + +**Dune** (Req 2.2): `dune tools path ` returns the executable path for a +specific tool. Per-tool granularity enables editors to find exact binaries. + +**uv**: ✓ `uv tool dir --bin` returns the bin directory (e.g., `~/.local/bin`). +`uv tool list` shows installed tools. No per-tool path command; construct path +as `$(uv tool dir --bin)/`. ([uv docs]) + +**cargo**: ✗ No discovery command. Binaries in `~/.cargo/bin/` (or +`$CARGO_INSTALL_ROOT/bin`). `cargo install --list` shows installed crates but +not paths. Scripts use `which ` or hardcode `~/.cargo/bin/`. ([cargo +docs]) + +**cargo-run-bin**: ✗ Binaries cached in `.bin/` directory. No CLI path command; +library API (`binary::install()`) returns paths programmatically. +([cargo-run-bin]) + +**npm**: ✓ `npm bin` returns `node_modules/.bin` directory (removed in npm 9+). +npm scripts automatically get `.bin` in PATH. No per-tool path command; use +`$(npm bin)/` or `./node_modules/.bin/`. ([npm docs: bin]) + +#### Tool upgrade + +Update installed tools to newer versions, optionally preserving version +constraints. + +**Dune** (Non-req 6.2): ✗ No upgrade command. Re-run `dune tools add ` to +get latest version, or `dune tools add .` for specific version. +Constraints from `(tool)` stanza preserved on re-lock. + +**uv**: ✓ `uv tool upgrade ` or `--all`. Respects original constraints: if +installed with `black>=23,<24`, upgrade stays within that range. To change +constraints, must reinstall with `uv tool install`. ([uv docs]) + +**cargo**: ✓\* `cargo install ` auto-detects newer versions without +`--force`. No constraint preservation; always installs latest (or specified) +version. Third-party `cargo-update` provides `cargo install-update -a` for batch +upgrades. ([cargo docs]) + +**cargo-run-bin**: ✗ No upgrade command. Edit version in `Cargo.toml` manually, +then run `cargo bin --install`. Version-controlled config ensures team sync. +([cargo-run-bin]) + +**npm**: ✓ `npm update` upgrades within semver constraints (e.g., `^1.2.3` +allows minor/patch updates). `npm outdated` shows available updates. For major +version bumps beyond constraints, use third-party `npm-check-updates`. ([npm +docs: update]) + +#### System-wide install + +Install tools globally, available outside any specific project. + +**Dune**: ✗ Intentionally omitted. Tools are workspace-local only. (See +[Non-requirement 6.1](#61-system-wide-installation)) + +**uv**: ✓ `uv tool install` adds to PATH. "Executables are installed to the uv +tools directory." ([uv docs]) + +**cargo**: ✓ `cargo install` puts binaries in `~/.cargo/bin/`, typically on +PATH. ([cargo docs]) + +**cargo-run-bin**: ✗ Project-local only. + +**npm**: ✓ `npm install -g` installs globally to system. ([npm docs]) + +### Recommendations from Research + +CR-soon Alizter: Review the entire Comparison with Other Tools section for +accuracy and completeness. Verify citations are still valid and that the +comparison table reflects current design decisions. + +**Adopted in this design**: + +- **Batch install** (from cargo-run-bin, npm): `dune tools add` with no + arguments locks all tools from `(tool)` stanzas (Req 4.1) +- **Declarative config** (from npm, cargo-run-bin): `(tool)` stanza in + `dune-workspace` (Req 4.2) +- **Binary selection** (from cargo-run-bin): `(executable ...)` field and + `--bin` flag (Req 1.6) + +**Future consideration**: + +- **`--locked` flag** (from cargo-run-bin): Ensure exact versions from solver, + not just constraints +- **Ephemeral run mode** (from uv's `uvx`): See + [Non-requirement 6.6](#66-ephemeral-runs) +- **System-wide linking** (from uv): See + [Non-requirement 6.1](#61-system-wide-installation) + +[uv docs]: https://docs.astral.sh/uv/concepts/tools/ +[uv docs: uvx]: https://docs.astral.sh/uv/concepts/tools/#running-tools +[cargo docs]: https://doc.rust-lang.org/cargo/commands/cargo-install.html +[cargo#5120]: https://github.com/rust-lang/cargo/issues/5120 +[rustup docs]: https://rust-lang.github.io/rustup/overrides.html +[cargo-run-bin]: https://github.com/dustinblackman/cargo-run-bin +[npm docs]: https://docs.npmjs.com/cli/ +[npm docs: folders]: https://docs.npmjs.com/cli/v9/configuring-npm/folders +[npm docs: package.json]: + https://docs.npmjs.com/cli/v10/configuring-npm/package-json +[npm docs: bin]: https://docs.npmjs.com/cli/v8/commands/npm-bin +[npm docs: update]: https://docs.npmjs.com/cli/v10/commands/npm-update +[npx docs]: https://docs.npmjs.com/cli/v10/commands/npx +[npm v7 changes]: + https://blog.npmjs.org/post/626173315965468672/npm-v7-series-beta-release-and-semver-major.html + +## Learnings from the Prototype + +CR-soon Alizter: This entire section may be considered garbage. Extract +important information out into the spec or remove entirely. + +This section documents key insights from the initial prototype implementation. + +### What Worked Well + +1. **Versioned lock directories**: Storing tools in + `_build/.tools.lock///` cleanly separates versions and keeps + the source tree clean. The flow (solve → extract version → move to versioned + path) is straightforward. + +2. **Tool isolation via synthetic wrapper**: Solving tools with a synthetic + `_tool_wrapper` package (not mixed with project packages) prevents + "packages outside workspace" solver errors. + +3. **Install cookie as build target**: Depending on `target/cookie` rather than + individual binaries correctly handles directory targets. The cookie is + created after full installation, providing a reliable completion signal. + +4. **Disk reads for lock checks**: Reading lock directories directly from disk + (not via build system) avoids triggering expensive project builds during tool + commands. + +5. **Simple PATH-only environment**: Tools only need the bin directory in PATH, + not a full exported environment. This avoids expensive closure computations. + +6. **Unified resolution with fallback**: A single resolution chain (stanza → + locked → PATH) provides clear priority and graceful degradation. + +### Problems Discovered + +1. **Directory targets cannot be depended upon directly**: Naively depending on + `_build/.../target/bin/` fails. Solution: depend on cookie, then read + cookie to discover binaries. + +2. **Build system triggers on lock checks**: Calling `Lock_dir.get` triggered + full builds. Solution: use `Lock_dir.read_disk` for checking. + +CR-soon Alizter: Need to understand what this is about. + +3. **Checksum collection missed versioned paths**: Fetch rules initially only + collected unversioned paths. Solution: scan `.tools.lock/` for all versions. + +4. **`.pkg/` rules gated by project lock**: Initial design skipped rule + generation when no project lock existed, breaking tools-only projects. + Solution: always generate rules; create tools-only DB when needed. + +5. **Compiler matching requires explicit package**: Just constraining `ocaml` + version wasn't enough; solver picked `ocaml-base-compiler` instead of + `ocaml-system`. Solution: explicitly require `ocaml-system` when system + compiler is available. + +### Reference Packages for Testing + +The implementation should be tested against these packages. We don't +special-case them, but they represent common tool patterns and historic +dev-tools support. + +CR-soon Alizter: Vet this list. Are there other common tools to add? Any that +should be removed? Consider tools from opam-repository's most-installed +packages. + +**Historic dev-tools** (from `src/dune_pkg/dev_tool.ml`): + +| Package | Binary | Notes | +| ------------------ | ---------------- | --------------------------------- | +| `ocamlformat` | `ocamlformat` | Also has `ocamlformat-rpc` binary | +| `odoc` | `odoc` | Documentation generator | +| `ocaml-lsp-server` | `ocamllsp` | Binary name differs from package | +| `utop` | `utop` | REPL, needs project libraries | +| `earlybird` | `ocamlearlybird` | Debugger | +| `odig` | `odig` | Documentation viewer | +| `opam-publish` | `opam-publish` | Release tooling | +| `dune-release` | `dune-release` | Release tooling | +| `ocaml-index` | `ocaml-index` | Indexer for ocamllsp | +| `merlin` | `ocamlmerlin` | Editor support | + +CR-soon Alizter: These are not historic dev tools, these are dev tools that we +currently support and would like to continue to support. Linking this to the +migration requirement. They will serve as test cases sure, but they also serve +as platform roadmap integration etc. -## TODO: Comparison with other tools +**Edge cases to verify**: -- uv -- cargo -- ?? +- Package with single binary: auto-selection works +- Package with multiple binaries: error with helpful message +- Package name ≠ binary name: `--bin` or `(executable)` works +- Tool needing project context: `utop` with `#require` -## TODO: Learnings from the initial dune prototype +CR-soon Alizter: I don't like these verifications here, they are out of place From 6b1d9d37415daa9370c1c3351a1344b71fe57816 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 00:41:33 -0400 Subject: [PATCH 06/72] Fix summary - It shouldn't mention implementation details (switch etc.) - It is too constrained (excluding tools defined in a project build) - Given unnecessary context (mentioning current design) - Does not include wide enough examples of tools (only mentions tools we widely used and already supported with bespoke mechanisms) Signed-off-by: Shon Feder --- doc/dev/tools.md | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index c28fbc3eb60..f1ad0a26d87 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -2,20 +2,10 @@ Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) -This document specifies dune's tool management system for development tools like -ocamlformat, ocamllsp, and odoc. - ## Summary -Tools are executables you use during development but that aren't part of your -project's build output. Think of each tool as an invisible opam switch: -independently solved, with its own dependency closure, isolated from your -project. - -**Why a new system?** The legacy dev-tools system only supported a hardcoded -list of tools and required dune package management. The new system works with -any opam package, whether you use dune's package management, an opam switch, or -system OCaml. +This document specifies dune's tool management system for tools like +ocamlformat, ocamllsp, odoc, or elpi. **Design principles:** From b32704c974f612ffdab8711af75e0cb60da96d27 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 00:44:08 -0400 Subject: [PATCH 07/72] Make design principles its own section Signed-off-by: Shon Feder --- doc/dev/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index f1ad0a26d87..caa6e94f85e 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -7,7 +7,7 @@ Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) This document specifies dune's tool management system for tools like ocamlformat, ocamllsp, odoc, or elpi. -**Design principles:** +## Design principles - **Generality**: Any opam package providing executables can be a tool - **Orthogonality**: Tools are solved independently from project dependencies From 4667076d8f5f9c3bfba49b62be96a20d97c8815b Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 00:44:57 -0400 Subject: [PATCH 08/72] Recover terminology This is essential for clear communication, otherwise key terms get used inconsistently or understood ambiguously. As a result of recovering terminology we also detect an error in the "generality" principle: packages themselves are not tools, but they provide tools so "any opam package ... can be a tool" is a category mistake. Signed-off-by: Shon Feder --- doc/dev/tools.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index caa6e94f85e..d474c22fc3c 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -7,9 +7,15 @@ Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) This document specifies dune's tool management system for tools like ocamlformat, ocamllsp, odoc, or elpi. +## Terminology + +- A **tool** is just an executable provided by some opam package. +- A tool is **available** if it is part of a well formed opam package obtainable + from any source: opam repository, pinning from a source, or defined locally. + ## Design principles -- **Generality**: Any opam package providing executables can be a tool +- **Generality**: Any executable defined in an opam packages can be a tool. - **Orthogonality**: Tools are solved independently from project dependencies - **Clean source tree**: All tool artifacts live in `_build/`, not your repo - **Multi-version**: Different projects can use different tool versions From 3c72d15d07622369fe8bbebdf134ddffa42572de Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 00:48:14 -0400 Subject: [PATCH 09/72] Remove non-requirements section This is not a useful section: things that aren't required just don't need to be listed as requirements. The canonical way of differentiating necessary from merely recommended requirements is MUST vs. SHOULD (or RECOMMENDED): https://www.ietf.org/rfc/rfc2119.txt Signed-off-by: Shon Feder --- doc/dev/tools.md | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index d474c22fc3c..b1c8108f7eb 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -27,7 +27,6 @@ Organized by category (installation, usability, dependencies, UI). Requirements describe user-facing behavior without specifying syntax or implementation. Cross-references point to the relevant Specification sections. -**Non-requirements** explicitly lists what's out of scope and why. **Specification** details _how_ the system implements the requirements: stanza syntax, CLI commands, version resolution algorithm, and directory structure. From 8240b4db1d5f1d39d64cbffef587c916ecdc85bd Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 00:56:43 -0400 Subject: [PATCH 10/72] Fix design principles If we are going to have a section of principles, let's make them principled principles. - "Clean source tree" is a requirement or an implementation detail - "Orthogonality" as stated was an implementation detail, generalizing it gives a principle. - "Multi-version" looks like a requirement, and it is entailed by a the principle of orthogonality Signed-off-by: Shon Feder --- doc/dev/tools.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index b1c8108f7eb..309ec59bb45 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -16,9 +16,8 @@ ocamlformat, ocamllsp, odoc, or elpi. ## Design principles - **Generality**: Any executable defined in an opam packages can be a tool. -- **Orthogonality**: Tools are solved independently from project dependencies -- **Clean source tree**: All tool artifacts live in `_build/`, not your repo -- **Multi-version**: Different projects can use different tool versions +- **Orthogonality**: Tool management should not interfere with other systems, + including the management of other tools. ## How to Read This Document From 5f8240a565d04a8d121bcb9a1fcfc8cee7ac2fdb Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:05:13 -0400 Subject: [PATCH 11/72] Consolidate and fix terminology - Drop implementation details from definition of "install" - Redefine "available" in terms of "install" as "installable" Signed-off-by: Shon Feder --- doc/dev/tools.md | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 309ec59bb45..154c69c105d 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -10,7 +10,8 @@ ocamlformat, ocamllsp, odoc, or elpi. ## Terminology - A **tool** is just an executable provided by some opam package. -- A tool is **available** if it is part of a well formed opam package obtainable +- To **install** a tool is to make the executable available within the workspace. +- A tool is **installable** if it is part of a well formed opam package obtainable from any source: opam repository, pinning from a source, or defined locally. ## Design principles @@ -174,18 +175,6 @@ requirements in However, wherever the roadmap specifies implementation details, we are free to deviate if needed to better satisfy the requirements. -### Terminology - -**Tool**: An executable provided by an opam package, used during development but -not part of the project's build output. Examples: ocamlformat (formatter), -ocamllsp (language server), odoc (documentation generator), utop (REPL). - -**Install**: Make a tool available for execution in the workspace. Unlike global -installation (`opam install`), tools are workspace-local. See -[CLI commands](#cli-commands) and [Directory structure](#directory-structure). - -CR-soon Alizter: Ephemeral could be explained here, its a bit obscure - ### 1. Installation Users must be able to install tools via Dune. From 2692a5e9d212e67f8ec2e23f05ec0cf9cb725e35 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:10:20 -0400 Subject: [PATCH 12/72] Fix section order - Everything but summary after TOC - "how to read doc" sections should be first Signed-off-by: Shon Feder --- doc/dev/tools.md | 75 ++++++++++++++++++++++++------------------------ 1 file changed, 38 insertions(+), 37 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 154c69c105d..b648d8f7c58 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -7,41 +7,6 @@ Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) This document specifies dune's tool management system for tools like ocamlformat, ocamllsp, odoc, or elpi. -## Terminology - -- A **tool** is just an executable provided by some opam package. -- To **install** a tool is to make the executable available within the workspace. -- A tool is **installable** if it is part of a well formed opam package obtainable - from any source: opam repository, pinning from a source, or defined locally. - -## Design principles - -- **Generality**: Any executable defined in an opam packages can be a tool. -- **Orthogonality**: Tool management should not interfere with other systems, - including the management of other tools. - -## How to Read This Document - -**Requirements** defines _what_ capabilities the system must provide and _why_. -Organized by category (installation, usability, dependencies, UI). Requirements -describe user-facing behavior without specifying syntax or implementation. -Cross-references point to the relevant Specification sections. - - -**Specification** details _how_ the system implements the requirements: stanza -syntax, CLI commands, version resolution algorithm, and directory structure. -Each section notes which requirements it implements. Open questions are marked -where decisions are pending. - -**Relationship to Package Management** explains the orthogonality principle and -how tools differ from project dependencies. - -**Comparison with Other Tools** analyzes how uv, cargo, cargo-run-bin, and npm -handle tool management, informing our design decisions. - -**CR markers** indicate areas needing further work: CR-soon for items to address -before stable release, CR-someday for post-v1 considerations. - @@ -52,8 +17,9 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [Tools](#tools) - [Summary](#summary) - [How to Read This Document](#how-to-read-this-document) + - [Terminology](#terminology) + - [Design principles](#design-principles) - [Requirements](#requirements) - - [Terminology](#terminology) - [1. Installation](#1-installation) - [1.1. Generality](#11-generality) - [1.2. Workspace-local](#12-workspace-local) @@ -167,6 +133,41 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni +## How to Read This Document + +**Requirements** defines _what_ capabilities the system must provide and _why_. +Organized by category (installation, usability, dependencies, UI). Requirements +describe user-facing behavior without specifying implementation details. +Cross-references point to the relevant Implementation sections. + +**Implementation** details _how_ the system implements the requirements: +stanza syntax, CLI commands, version resolution algorithm, and directory +structure. Each section notes which requirements it implements. Open questions +are marked where decisions are pending. + +**Relationship to Package Management** explains the orthogonality principle and +how tools differ from project dependencies. + +**Comparison with Other Tools** analyzes how uv, cargo, cargo-run-bin, and npm +handle tool management, informing our design decisions. + +**CR markers** indicate areas needing further work: CR-soon for items to address +before stable release, CR-someday for post-v1 considerations. + +## Terminology + +- A **tool** is just an executable provided by some opam package. +- To **install** a tool is to make the executable available within the workspace. +- A tool is **installable** if it is part of a well formed opam package obtainable + from any source: opam repository, pinning from a source, or defined locally. + +## Design principles + +- **Generality**: Any executable defined in an opam packages can be a tool. +- **Orthogonality**: Tool management should not interfere with other systems, + including the management of other tools. + + ## Requirements NOTE: The requirements here should be compatible with the functional @@ -618,7 +619,7 @@ CR-soon Alizter: Mention that autolocking makes this somewhat feasable, but it c be relied on as it will roll to the newer versions. There are no binary installs so therefore it would be slow also -## Specification +## Implementation ### The `(tool)` stanza From 8124645cd4cd43a6c171fa924c27b41db5a13ce7 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:16:49 -0400 Subject: [PATCH 13/72] Fixes to how to read - May as well link the sections. - Explain what the terminology section is for. - Remove CR explanation: no place for those in the mergable doc. Signed-off-by: Shon Feder --- doc/dev/tools.md | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index b648d8f7c58..7fdbdf24cdc 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -135,24 +135,27 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni ## How to Read This Document -**Requirements** defines _what_ capabilities the system must provide and _why_. -Organized by category (installation, usability, dependencies, UI). Requirements -describe user-facing behavior without specifying implementation details. -Cross-references point to the relevant Implementation sections. +[Terminology](#terminology) defines key terms used throughout the document. +Precise definition of technical terms prevents misunderstandings and certain +classes of requirement error. -**Implementation** details _how_ the system implements the requirements: +[Requirements](#requirements) defines _what_ capabilities the system must +provide and _why_. Organized by category. Requirements describe user-facing +behavior without specifying implementation details. Cross-references point to +the relevant Implementation sections. + +[Implementation](#implementation) details _how_ the system implements the requirements: stanza syntax, CLI commands, version resolution algorithm, and directory structure. Each section notes which requirements it implements. Open questions are marked where decisions are pending. -**Relationship to Package Management** explains the orthogonality principle and -how tools differ from project dependencies. - -**Comparison with Other Tools** analyzes how uv, cargo, cargo-run-bin, and npm -handle tool management, informing our design decisions. +[Relationship to Package Management](#relationship-to-package-management) +explains the orthogonality principle and how tools differ from project +dependencies. -**CR markers** indicate areas needing further work: CR-soon for items to address -before stable release, CR-someday for post-v1 considerations. +[Comparison with Other Tools](comparison-with-other-tools) analyzes how uv, +cargo, cargo-run-bin, and npm handle tool management, informing our design +decisions. ## Terminology From 63809e722157097b870d215dd55b1d38fd6374fb Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:18:57 -0400 Subject: [PATCH 14/72] Explain design principles Signed-off-by: Shon Feder --- doc/dev/tools.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 7fdbdf24cdc..901776b7d5b 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -139,6 +139,9 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni Precise definition of technical terms prevents misunderstandings and certain classes of requirement error. +[Design principles](#design-principles) specifies the guiding principles +informing the design. + [Requirements](#requirements) defines _what_ capabilities the system must provide and _why_. Organized by category. Requirements describe user-facing behavior without specifying implementation details. Cross-references point to @@ -170,7 +173,6 @@ decisions. - **Orthogonality**: Tool management should not interfere with other systems, including the management of other tools. - ## Requirements NOTE: The requirements here should be compatible with the functional From cc08d27fa04b50906fbc0f9f12b47bedb7811f19 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:24:21 -0400 Subject: [PATCH 15/72] Define a well formed opam package Signed-off-by: Shon Feder --- doc/dev/tools.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 901776b7d5b..6ddec20fca1 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -164,6 +164,8 @@ decisions. - A **tool** is just an executable provided by some opam package. - To **install** a tool is to make the executable available within the workspace. +- A **well formed opam package** specifies all data necessary to install and + build its provided targets. - A tool is **installable** if it is part of a well formed opam package obtainable from any source: opam repository, pinning from a source, or defined locally. From ae82a64964eef7395d03ea383eaa7dd2d91e589b Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:25:34 -0400 Subject: [PATCH 16/72] Fix roadmap reference Make it clear that, while compatability is required, satisfaction is not. Signed-off-by: Shon Feder --- doc/dev/tools.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 6ddec20fca1..91e1ec2c78a 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -181,7 +181,8 @@ NOTE: The requirements here should be compatible with the functional requirements in [https://ocaml.org/tools/platform-roadmap](https://ocaml.org/tools/platform-roadmap). However, wherever the roadmap specifies implementation details, we are free to -deviate if needed to better satisfy the requirements. +deviate if needed to better satisfy the requirements, and our task here is not +to realize all of the requirements in the platform roadmap. ### 1. Installation From f346584e99ed00fdebc8123f4a95dfd065438781 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:27:09 -0400 Subject: [PATCH 17/72] Fix generality Reference to legacy system is not relevant for requirements (could be in implementation) Signed-off-by: Shon Feder --- doc/dev/tools.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 91e1ec2c78a..aa079bc3d00 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -190,9 +190,7 @@ Users must be able to install tools via Dune. #### 1.1. Generality -Any opam package that provides executables can be used as a tool. Unlike the -legacy system, tools are not hardcoded into dune. Users can install arbitrary -packages without waiting for dune releases. +Any installable tool must be supported. #### 1.2. Workspace-local From 23205d1681df2890e200ceba1ff7b5b8587b7ee5 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:30:54 -0400 Subject: [PATCH 18/72] rm outdated CR Signed-off-by: Shon Feder --- doc/dev/tools.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index aa079bc3d00..daaf4228b4a 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -199,10 +199,6 @@ isolated tool installations that don't affect other workspaces or the system. See [Directory structure](#directory-structure) for storage locations. -CR-soon Alizter: Once we have the relocatable compiler, tools being built will -be cached so they aren't "installed". They will materialise in \_build/ but that -shouldn't be a persistent store. - CR-someday Alizter: Consider supporting `(tool)` stanzas in `dune-project` files for per-project tool configuration, similar to how `(pin)` exists in both `dune-workspace` and `dune-project`. This would allow different projects within From f649f375c31b2b83c9ced100e30fb138f49220b1 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:31:10 -0400 Subject: [PATCH 19/72] Move CR into "Ideas" section These are design ideas for possible functionality, they are not relevant to functional specifications. Signed-off-by: Shon Feder --- doc/dev/tools.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index daaf4228b4a..0c0b8f08d34 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -199,15 +199,6 @@ isolated tool installations that don't affect other workspaces or the system. See [Directory structure](#directory-structure) for storage locations. -CR-someday Alizter: Consider supporting `(tool)` stanzas in `dune-project` files -for per-project tool configuration, similar to how `(pin)` exists in both -`dune-workspace` and `dune-project`. This would allow different projects within -a workspace to declare different tool requirements. Conflict resolution: -workspace wins over project, outer project wins over nested. Sibling projects -with conflicting declarations can coexist. Within project-scoped rules the -correct tool is unambiguous. CLI commands without project context (e.g., -`dune tools run` from workspace root) should error if declarations conflict. - #### 1.3. Version pinning Users must be able to install specific versions of tools via: @@ -2001,3 +1992,14 @@ as platform roadmap integration etc. - Tool needing project context: `utop` with `#require` CR-soon Alizter: I don't like these verifications here, they are out of place + +## Ideas + +CR-someday Alizter: Consider supporting `(tool)` stanzas in `dune-project` files +for per-project tool configuration, similar to how `(pin)` exists in both +`dune-workspace` and `dune-project`. This would allow different projects within +a workspace to declare different tool requirements. Conflict resolution: +workspace wins over project, outer project wins over nested. Sibling projects +with conflicting declarations can coexist. Within project-scoped rules the +correct tool is unambiguous. CLI commands without project context (e.g., +`dune tools run` from workspace root) should error if declarations conflict. From f418f7f03bcfe0eafb9e7530e11c418d34bba94f Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:44:52 -0400 Subject: [PATCH 20/72] Recover system wide req But make this a "SHOULD" rather than a "MUST" Signed-off-by: Shon Feder --- doc/dev/tools.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 0c0b8f08d34..b5fc2b0c828 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -199,7 +199,19 @@ isolated tool installations that don't affect other workspaces or the system. See [Directory structure](#directory-structure) for storage locations. -#### 1.3. Version pinning +#### 1.3. System wide + +Users should be able to install tools in a way that allows them to be used in the +system-wide environment (e.g., outside of any particular sandbox). + +*NOTE:* This does not dictate that dune must maintain the equivalent of +default switches, or predetermine any other implementation choice. But the +support for tool management must be designed in way that makes it simple and +reliable for users to use installed tools outside of a project sandbox (e.g., by +adding the location of a directory of binaries to their `PATH` or some other +means). + +#### 1.4. Version specification Users must be able to install specific versions of tools via: From 55e076511017a63d848a12169028f85787b7f227 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 01:48:36 -0400 Subject: [PATCH 21/72] Move idea Signed-off-by: Shon Feder --- doc/dev/tools.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index b5fc2b0c828..78a5ad776e8 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -222,25 +222,22 @@ Users must be able to install specific versions of tools via: See [Version syntax](#version-syntax) for CLI syntax and [The `(tool)` stanza](#the-tool-stanza) for declarative configuration. -CR-soon Alizter: Edge cases to specify: + +##### 1.4.1. Version consistency + +Version specifications provided via CLI must override persistent - Stanza says `(= 0.26.2)` but CLI requests `0.27.0`: which wins? - `.ocamlformat` says `version=0.26.2` but stanza says `(= 0.27.0)`: conflict? - Version doesn't exist in opam-repository: error message? -#### 1.4. Multi-version support +#### 1.5. Multi-version support Multiple versions of the same tool can coexist within a workspace. This enables per-project tool resolution - for example, different projects within a workspace can have different `.ocamlformat` files specifying different versions, and the formatting rules will use the correct version for each project. -CR-someday Alizter: Multi-version scaling. Supporting N versions means N -separate solves, downloads, and builds. In a monorepo migrating ocamlformat -versions across 50 sub-projects, users could accumulate many versions. Consider: -(1) warning when version count exceeds threshold, (2) `dune tools gc` to remove -unused versions, (3) documenting expected steady-state (few versions, not many). - See [Tool pforms](#tool-pforms) for how build rules can reference versioned tool executables. @@ -2015,3 +2012,9 @@ workspace wins over project, outer project wins over nested. Sibling projects with conflicting declarations can coexist. Within project-scoped rules the correct tool is unambiguous. CLI commands without project context (e.g., `dune tools run` from workspace root) should error if declarations conflict. + +CR-someday Alizter: Multi-version scaling. Supporting N versions means N +separate solves, downloads, and builds. In a monorepo migrating ocamlformat +versions across 50 sub-projects, users could accumulate many versions. Consider: +(1) warning when version count exceeds threshold, (2) `dune tools gc` to remove +unused versions, (3) documenting expected steady-state (few versions, not many). From 4a52a6285e81ff4cf8687114d2c69f3832065bda Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 02:01:27 -0400 Subject: [PATCH 22/72] Resolve binary selection Since packages provide tools, rather than define them, we avoid equivocating between package names and tools: a package may provide a tool that shares the package name, but when tools are invoked, it is done by the name of tool executable, not by the name of the package. Therefore, binary selection is just the ability to filter a subset of available packages in case you don't want everything provided by a package (the default). Signed-off-by: Shon Feder --- doc/dev/tools.md | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 78a5ad776e8..be66ff81c32 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -259,25 +259,14 @@ and becomes fast with full caching. #### 1.6. Binary selection -When a tool package provides multiple executables, users must be able to: - -- Specify which binary to run (per-invocation) -- Configure a default binary (persistent) - -When a package provides a single binary, it should be selected automatically. +When a package provides multiple tools, users must be able to specify a subset +for installation. When a package providing tools is installed without +qualification, all provided tools must be installed. As a special case, when a +package provides a single binary, it will be installed without need to qualify. See [The `(tool)` stanza](#the-tool-stanza) and [CLI commands](#cli-commands) for syntax. -CR-soon Alizter: Here is an annoyance: opam packages don't specify which -binaries they install, which is fine. Once we have built the package we can -inspect which binaries we have. For something like `ocaml-lsp-server` this will -be the `ocamllsp` binary. For `ocamlformat` there are two: `ocamlformat` which -is intended and `ocamlformat-rpc` which is likely not, in this case the -prototype errors and asks the user via the CLI to be more specific by providing -the `--bin` arg. We should evaluate if this is going to be the best way to solve -this issue. - ### 2. Usability Users must be able to run tools installed by Dune. From 8aff54d5ae84c970c88824da827abfc1751a4983 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 02:03:44 -0400 Subject: [PATCH 23/72] Open problem on version consistency Signed-off-by: Shon Feder --- doc/dev/tools.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index be66ff81c32..b0975c4f428 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -222,15 +222,18 @@ Users must be able to install specific versions of tools via: See [Version syntax](#version-syntax) for CLI syntax and [The `(tool)` stanza](#the-tool-stanza) for declarative configuration. - -##### 1.4.1. Version consistency +##### 1.4.1. Version consistency TODO -Version specifications provided via CLI must override persistent +What happens in case of conflict? - Stanza says `(= 0.26.2)` but CLI requests `0.27.0`: which wins? - `.ocamlformat` says `version=0.26.2` but stanza says `(= 0.27.0)`: conflict? - Version doesn't exist in opam-repository: error message? +CR Shon: could we require consistency rather than needing to support precedence? +E.g., setting the version by CLI would cause an update to any config file, and +config files must be kept up to date? + #### 1.5. Multi-version support Multiple versions of the same tool can coexist within a workspace. This enables From 03615f5ee123fd80364993b09d2a8ee27b978f6d Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 02:09:08 -0400 Subject: [PATCH 24/72] Recover shells requirement No idea why this was merged alongside an unrelated requirement. Factor the versions pecification requirement into its own issue. Signed-off-by: Shon Feder --- doc/dev/tools.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index b0975c4f428..c942e82fc92 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -274,12 +274,15 @@ for syntax. Users must be able to run tools installed by Dune. -#### 2.1. Tool invocation +#### 2.1. Shells -Users must be able to: +Users must be able to run tools by invoking them directly in any shell (e.g., bash). -- Run installed tools from any shell -- Request a specific version when multiple versions are installed +See [CLI commands](#cli-commands) for invocation syntax. + +#### 2.2. Version specification TODO + +Users must be able run a specific version of a tool when multiple versions are installed. See [CLI commands](#cli-commands) for invocation syntax. @@ -290,7 +293,7 @@ CR-soon Alizter: Edge cases to specify: - Specified version not locked: error with suggestion? - Build fails: propagate build error? -#### 2.2. Programmatic use +#### 2.3. Programmatic use Other programs (e.g., editor plugins, scripts) must be able to programmatically discover the path to a tool's executable without parsing internal directory @@ -320,12 +323,12 @@ simplifies editor integration (one directory to add to PATH), avoids PATH length limits (especially on Windows), and provides a single stable location for discovery. Trade-off: need to maintain symlinks as tools are added/removed. -#### 2.3. Dune subcommands +#### 2.4. Dune subcommands Dune subcommands that invoke external tools must be able to use tools managed by `dune tools`. See [Dune Integration](#5-dune-integration). -#### 2.4. System PATH fallback +#### 2.5. System PATH fallback When a tool is not locked and no version is specified (e.g., `.ocamlformat` without a version), dune subcommands should fall back to the system PATH. @@ -340,7 +343,7 @@ CR-Alizter soon: Fixup wording above. CR-Alizter soon: How would this even work if we don't know the binary from the package? -#### 2.5. Editor integration +#### 2.6. Editor integration Editors and IDEs must be able to: From ca703e08175d6319d7aabed976c8920677d38164 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 26 May 2026 02:23:23 -0400 Subject: [PATCH 25/72] Fix programatic use Not sure why this overspecification and reduction in generality. We want the general usability. Signed-off-by: Shon Feder --- doc/dev/tools.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index c942e82fc92..69f52503ef2 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -295,14 +295,13 @@ CR-soon Alizter: Edge cases to specify: #### 2.3. Programmatic use -Other programs (e.g., editor plugins, scripts) must be able to programmatically -discover the path to a tool's executable without parsing internal directory -structures. +Other programs (e.g., editor plugins) must be able to find and run installed +tools via a single, transparent mechanism (e.g., an equivalent to `opam env` or +by adding a single directory of executables to the lookup path, or some other +means). See [CLI commands](#cli-commands) for the discovery interface. -CR-soon Alizter: To be clear, this is replacing the `dune tools which ` -that we currently have. Not sure if its the best design. CR-soon Alizter: `dune tools path` behavior is underspecified: @@ -1429,6 +1428,12 @@ our design decisions and identifying gaps in our approach. **Relevance to requirements**: +- 2.1: uv support adding its binary directory to shells https://docs.astral.sh/uv/concepts/tools/#tool-executables +- 2.2: + - uv uses a simple, predictable location for binary installations + https://docs.astral.sh/uv/reference/storage/#tool-executables and it allows + - uv allows configuration of the installation path and an API for locating the + bin directory https://docs.astral.sh/uv/reference/storage/#tool-executables - **3.2 Dependency isolation**: Per-tool venvs achieve isolation. Our separate lock dirs provide similar isolation. From 19ed74045478be55c6abc9cc2ac5f78da8ef5ba4 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Mon, 6 Jul 2026 22:57:46 -0400 Subject: [PATCH 26/72] Update toc Signed-off-by: Shon Feder --- doc/dev/tools.md | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 69f52503ef2..043683bf003 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -23,16 +23,19 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [1. Installation](#1-installation) - [1.1. Generality](#11-generality) - [1.2. Workspace-local](#12-workspace-local) - - [1.3. Version pinning](#13-version-pinning) - - [1.4. Multi-version support](#14-multi-version-support) + - [1.3. System wide](#13-system-wide) + - [1.4. Version specification](#14-version-specification) + - [1.4.1. Version consistency TODO](#141-version-consistency-todo) + - [1.5. Multi-version support](#15-multi-version-support) - [1.5. Clean source tree](#15-clean-source-tree) - [1.6. Binary selection](#16-binary-selection) - [2. Usability](#2-usability) - - [2.1. Tool invocation](#21-tool-invocation) - - [2.2. Programmatic use](#22-programmatic-use) - - [2.3. Dune subcommands](#23-dune-subcommands) - - [2.4. System PATH fallback](#24-system-path-fallback) - - [2.5. Editor integration](#25-editor-integration) + - [2.1. Shells](#21-shells) + - [2.2. Version specification TODO](#22-version-specification-todo) + - [2.3. Programmatic use](#23-programmatic-use) + - [2.4. Dune subcommands](#24-dune-subcommands) + - [2.5. System PATH fallback](#25-system-path-fallback) + - [2.6. Editor integration](#26-editor-integration) - [3. Dependency interactions](#3-dependency-interactions) - [3.1. Compiler compatibility](#31-compiler-compatibility) - [3.2. Dependency isolation](#32-dependency-isolation) @@ -55,7 +58,7 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [6.4. Environment manipulation](#64-environment-manipulation) - [6.5. Precompiled binaries](#65-precompiled-binaries) - [6.6. Ephemeral runs](#66-ephemeral-runs) - - [Specification](#specification) + - [Implementation](#implementation) - [The `(tool)` stanza](#the-tool-stanza) - [Syntax](#syntax) - [Fields](#fields) @@ -130,6 +133,7 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [What Worked Well](#what-worked-well) - [Problems Discovered](#problems-discovered) - [Reference Packages for Testing](#reference-packages-for-testing) + - [Ideas](#ideas) @@ -295,14 +299,13 @@ CR-soon Alizter: Edge cases to specify: #### 2.3. Programmatic use -Other programs (e.g., editor plugins) must be able to find and run installed -tools via a single, transparent mechanism (e.g., an equivalent to `opam env` or -by adding a single directory of executables to the lookup path, or some other -means). +Programs (e.g., editor plugins) must be able to find and run installed +tools via a single, transparent mechanism (e.g., the `dune tools env` equivalent +to `opam env` or by adding a single directory of executables to the lookup path, +or some other means). See [CLI commands](#cli-commands) for the discovery interface. - CR-soon Alizter: `dune tools path` behavior is underspecified: - Should `path` trigger download and build if not yet built? Or only return path @@ -322,10 +325,10 @@ simplifies editor integration (one directory to add to PATH), avoids PATH length limits (especially on Windows), and provides a single stable location for discovery. Trade-off: need to maintain symlinks as tools are added/removed. -#### 2.4. Dune subcommands +##### 2.3.1 dune subcommands -Dune subcommands that invoke external tools must be able to use tools managed by -`dune tools`. See [Dune Integration](#5-dune-integration). +As a special case, dune subcommands that invoke external tools must be able to +use tools managed by `dune tools`. See [Dune Integration](#5-dune-integration). #### 2.5. System PATH fallback From 54f83805f51450a0fb073586e8c31b81db81fa79 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 7 Jul 2026 23:32:07 -0400 Subject: [PATCH 27/72] Remove "Non-requirements" section Many of there are out of date, e.g., work on in-and-out is onging, and we have settled in conversation that support for something like an `env` subcommand is required (most recently in the metting with the team working on editors). There is also no reason to rule out precompiled binaries or ephemeral runs: if we don't state those as requirments now, we won't work on implementing them at this point. There is no reason in principle such things shouldn't be considered in the future. Signed-off-by: Shon Feder --- doc/dev/tools.md | 98 ------------------------------------------------ 1 file changed, 98 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 043683bf003..9e5ee68b15e 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -520,104 +520,6 @@ CR-soon Alizter: Detail the migration story: - What happens to existing `.dev-tools.locks/` directories? - User-facing migration guide (re-add tools via `dune tools add`) -### 6. Non-requirements - -The following are explicitly out of scope: - -#### 6.1. System-wide installation - -Tools are workspace-local, not installed system-wide like `opam install` or -`cargo install`. There is no global tool registry outside of workspaces. - -Note: While tools are workspace-local, builds are still cached via the dune -build cache. If two workspaces install the same tool version with the same -compiler, the second workspace reuses the cached build artifacts. - -CR-someday Alizter: Consider system-wide tool installation for use outside -workspaces. Tools could be stored in `~/.cache/dune/tools/` and invoked via -`dune tools run` from any directory. This would enable: - -- Global utop without an opam switch -- Running OCaml scripts with dependencies (`dune run script.ml`) -- `dune shell` outside workspaces The same isolation and compiler-matching logic - applies. Related: [#12107] (system-wide packages). - -[#12107]: https://github.com/ocaml/dune/issues/12107 - -#### 6.2. Automatic updates - -Tools do not auto-update; explicit user action is required. - -#### 6.3. Tools for tools - -Using tools while developing the tools themselves (or their dependencies) has -nuances related to the "in and out" problem ([#8652]). - -**Different scenarios**: - -- **Working on a tool's dependency** (e.g., a library that ocamlformat uses): - You probably don't need ocamlformat to use your local changes - you just want - ocamlformat to work for formatting your code. The standard tool isolation - handles this fine. - -- **Working on the tool itself** (e.g., developing ocamllsp): You likely want to - use the ocamllsp you're building, not a separately installed version. This is - where pins could help - pin the tool to the local path. But circular builds - (tool needs to build itself) remain unsolved. - -The first scenario is common and supported. The second is out of scope until the -in-and-out problem is resolved, though pins may provide a partial workaround for -non-circular cases. - -#### 6.4. Environment manipulation - -No `dune tools env` command. Unlike `opam env`, we avoid stateful shell -environment manipulation. Use `dune tools run` or `dune tools path` instead. - -We acknowledge this adds friction for users migrating from opam and for -interactive shell use. However, we believe explicit invocation is preferable to -stateful environment that can silently become stale or inconsistent. - -CR-soon Alizter: There's a tension here: we reject `dune tools env` as too -implicit, but PATH fallback (2.4) is also implicit. Clarify the philosophy: is -the issue stateful shells specifically, or implicit behavior generally? - -CR-soon Alizter: dune tools env exists in the prototype, but it may be -undesirable. - -CR-someday Alizter: Consider `dune shell` as an alternative to `dune tools env`. -Instead of mutating the current shell's environment (opam-style), spawn a -subshell with tools in PATH (Nix-style). This avoids stale environment issues -and makes the boundary clear - just exit the subshell to return to normal. Could -also support `dune shell --with ocamlformat ocamllsp` syntax. - -#### 6.5. Precompiled binaries - -Tools are always built from source. Downloading precompiled binaries (like -cargo-binstall or uv's wheel downloads) is out of scope. - -This ensures compiler compatibility (Req 3.1) and avoids ABI (Application Binary -Interface) mismatch issues. OCaml binaries embed compiler-version-specific -formats - for example, `.cmi` files have checksums tied to compiler internals. A -pre-built tool (e.g., ocamllsp compiled with a different toolchain) cannot read -build artifacts from a differently-compiled project, causing "corrupted compiled -interface" errors ([#11229]). - -[#11229]: https://github.com/ocaml/dune/issues/11229 - -#### 6.6. Ephemeral runs - -No `uvx`-style temporary execution mode. Tools must be locked before running. - -Building OCaml tools from source is slow compared to downloading Python wheels, -so discarding the build after one use would be wasteful. Locking caches the -build for reuse. Users who want to try a tool temporarily can use -`dune tools add`, run it, then `dune tools remove`. - -CR-soon Alizter: Mention that autolocking makes this somewhat feasable, but it cannot -be relied on as it will roll to the newer versions. There are no binary -installs so therefore it would be slow also - ## Implementation ### The `(tool)` stanza From 9608f7d9b637dee69a039059bd72922b3e619611 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 7 Jul 2026 23:36:32 -0400 Subject: [PATCH 28/72] updat toc Signed-off-by: Shon Feder --- doc/dev/tools.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/doc/dev/tools.md b/doc/dev/tools.md index 9e5ee68b15e..bcb055cca24 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools.md @@ -33,7 +33,7 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [2.1. Shells](#21-shells) - [2.2. Version specification TODO](#22-version-specification-todo) - [2.3. Programmatic use](#23-programmatic-use) - - [2.4. Dune subcommands](#24-dune-subcommands) + - [2.3.1 dune subcommands](#231-dune-subcommands) - [2.5. System PATH fallback](#25-system-path-fallback) - [2.6. Editor integration](#26-editor-integration) - [3. Dependency interactions](#3-dependency-interactions) @@ -51,13 +51,6 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [5.3. REPL (`dune utop`, `dune ocaml utop`)](#53-repl-dune-utop-dune-ocaml-utop) - [5.4. Tool references in actions](#54-tool-references-in-actions) - [5.5. Legacy migration](#55-legacy-migration) - - [6. Non-requirements](#6-non-requirements) - - [6.1. System-wide installation](#61-system-wide-installation) - - [6.2. Automatic updates](#62-automatic-updates) - - [6.3. Tools for tools](#63-tools-for-tools) - - [6.4. Environment manipulation](#64-environment-manipulation) - - [6.5. Precompiled binaries](#65-precompiled-binaries) - - [6.6. Ephemeral runs](#66-ephemeral-runs) - [Implementation](#implementation) - [The `(tool)` stanza](#the-tool-stanza) - [Syntax](#syntax) From 8896f920ebc756a85726a398096eab8ab8ea66c3 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Wed, 8 Jul 2026 00:07:48 -0400 Subject: [PATCH 29/72] Split req and impl specs into different files It is important to be able to consider the requirements above the code level and without commitment to specific implementation details. Splitting the implementation spec notes into their own file help support this abstraction. Signed-off-by: Shon Feder --- .../implementation.md} | 582 +++--------------- doc/dev/tools-design/requirements.md | 437 +++++++++++++ 2 files changed, 518 insertions(+), 501 deletions(-) rename doc/dev/{tools.md => tools-design/implementation.md} (72%) create mode 100644 doc/dev/tools-design/requirements.md diff --git a/doc/dev/tools.md b/doc/dev/tools-design/implementation.md similarity index 72% rename from doc/dev/tools.md rename to doc/dev/tools-design/implementation.md index bcb055cca24..3087dbb0ecd 100644 --- a/doc/dev/tools.md +++ b/doc/dev/tools-design/implementation.md @@ -1,11 +1,10 @@ -# Tools +## Tools Implementation Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) ## Summary -This document specifies dune's tool management system for tools like -ocamlformat, ocamllsp, odoc, or elpi. +This document specifies the implementatino of dune's tool management system. **Table of Contents** -- [Tools](#tools) - - [Summary](#summary) - - [How to Read This Document](#how-to-read-this-document) - - [Terminology](#terminology) - - [Design principles](#design-principles) - - [Requirements](#requirements) - - [1. Installation](#1-installation) - - [1.1. Generality](#11-generality) - - [1.2. Workspace-local](#12-workspace-local) - - [1.3. System wide](#13-system-wide) - - [1.4. Version specification](#14-version-specification) - - [1.4.1. Version consistency TODO](#141-version-consistency-todo) - - [1.5. Multi-version support](#15-multi-version-support) - - [1.5. Clean source tree](#15-clean-source-tree) - - [1.6. Binary selection](#16-binary-selection) - - [2. Usability](#2-usability) - - [2.1. Shells](#21-shells) - - [2.2. Version specification TODO](#22-version-specification-todo) - - [2.3. Programmatic use](#23-programmatic-use) - - [2.3.1 dune subcommands](#231-dune-subcommands) - - [2.5. System PATH fallback](#25-system-path-fallback) - - [2.6. Editor integration](#26-editor-integration) - - [3. Dependency interactions](#3-dependency-interactions) - - [3.1. Compiler compatibility](#31-compiler-compatibility) - - [3.2. Dependency isolation](#32-dependency-isolation) - - [3.3. No build triggers](#33-no-build-triggers) - - [3.4. Watch mode integration](#34-watch-mode-integration) - - [3.5. Dog fooding](#35-dog-fooding) - - [4. UI](#4-ui) - - [4.1. CLI](#41-cli) - - [4.2. Persistent configuration](#42-persistent-configuration) - - [5. Dune Integration](#5-dune-integration) - - [5.1. Format rules (`dune fmt`, `dune build @fmt`)](#51-format-rules-dune-fmt-dune-build-fmt) - - [5.2. Documentation rules (`dune build @doc`, `dune ocaml doc`)](#52-documentation-rules-dune-build-doc-dune-ocaml-doc) - - [5.3. REPL (`dune utop`, `dune ocaml utop`)](#53-repl-dune-utop-dune-ocaml-utop) - - [5.4. Tool references in actions](#54-tool-references-in-actions) - - [5.5. Legacy migration](#55-legacy-migration) - - [Implementation](#implementation) - - [The `(tool)` stanza](#the-tool-stanza) - - [Syntax](#syntax) - - [Fields](#fields) - - [Compiler matching](#compiler-matching) - - [Examples](#examples) - - [Invalid tool selections](#invalid-tool-selections) - - [CLI commands](#cli-commands) - - [Version syntax](#version-syntax) - - [Batch operations](#batch-operations) - - [Open question: version separator](#open-question-version-separator) - - [Open question: `add` vs `install` naming](#open-question-add-vs-install-naming) - - [Open question: `dune tools run` PATH fallback](#open-question-dune-tools-run-path-fallback) - - [Tool resolution](#tool-resolution) - - [Source resolution](#source-resolution) - - [Version selection](#version-selection) - - [Example scenarios](#example-scenarios) - - [Directory structure](#directory-structure) - - [Tool pforms](#tool-pforms) - - [Syntax](#syntax-1) - - [Resolution](#resolution) - - [Examples](#examples-1) - - [Future: library access](#future-library-access) - - [Relationship to Package Management](#relationship-to-package-management) - - [Orthogonality: a key design principle](#orthogonality-a-key-design-principle) - - [The package-tool continuum](#the-package-tool-continuum) - - [Key differences from project dependencies](#key-differences-from-project-dependencies) - - [Why tools don't use `dune.lock/`](#why-tools-dont-use-dunelock) - - [Locking vs building](#locking-vs-building) - - [Future: Relocatable compiler and caching](#future-relocatable-compiler-and-caching) - - [Potential future stanza options](#potential-future-stanza-options) - - [GitHub Issues Addressed](#github-issues-addressed) - - [Arbitrary tool packages ([#12913])](#arbitrary-tool-packages-12913) - - [Lock directory clutter ([#10955], [#12097])](#lock-directory-clutter-10955-12097) - - [Compiler ABI mismatch ([#11229])](#compiler-abi-mismatch-11229) - - [Graceful fallback for missing tools ([#10578])](#graceful-fallback-for-missing-tools-10578) - - [Version constraints ([#12777])](#version-constraints-12777) - - [Version consistency ([#10688])](#version-consistency-10688) - - [Dev tools redesign ([#12914])](#dev-tools-redesign-12914) - - [Tool dependency isolation ([#12551])](#tool-dependency-isolation-12551) - - [No project build triggers ([#11037])](#no-project-build-triggers-11037) - - [Compiler-independent formatters ([#11038])](#compiler-independent-formatters-11038) - - [Version from config files ([#5315])](#version-from-config-files-5315) - - [Version constraint parsing ([#12866])](#version-constraint-parsing-12866) - - [Opam with-dev-setup integration ([#12135])](#opam-with-dev-setup-integration-12135) - - [Helpful error messages ([#13235], [#12975])](#helpful-error-messages-13235-12975) - - [Batch installation ([#12557])](#batch-installation-12557) - - [Failure isolation ([#12818])](#failure-isolation-12818) - - [Utop findlib integration ([#13471])](#utop-findlib-integration-13471) - - [Comparison with Other Tools](#comparison-with-other-tools) - - [uv (Python)](#uv-python) - - [cargo (Rust)](#cargo-rust) - - [cargo-run-bin (Rust community tool)](#cargo-run-bin-rust-community-tool) - - [npm (JavaScript)](#npm-javascript) - - [How Our Design Compares](#how-our-design-compares) - - [Tools compared](#tools-compared) - - [Comparison table](#comparison-table) - - [Project-local](#project-local) - - [Multi-version](#multi-version) - - [Declarative config](#declarative-config) - - [Compiler matching](#compiler-matching-1) - - [PATH fallback](#path-fallback) - - [Precompiled binaries](#precompiled-binaries) - - [Tool isolation](#tool-isolation) - - [Ephemeral runs](#ephemeral-runs) - - [Batch install](#batch-install) - - [Binary selection](#binary-selection) - - [Tool discovery](#tool-discovery) - - [Tool upgrade](#tool-upgrade) - - [System-wide install](#system-wide-install) - - [Recommendations from Research](#recommendations-from-research) - - [Learnings from the Prototype](#learnings-from-the-prototype) - - [What Worked Well](#what-worked-well) - - [Problems Discovered](#problems-discovered) - - [Reference Packages for Testing](#reference-packages-for-testing) - - [Ideas](#ideas) +- [Tools Implementation](#tools-implementation) +- [Summary](#summary) + - [The `(tool)` stanza](#the-tool-stanza) + - [Syntax](#syntax) + - [Fields](#fields) + - [Compiler matching](#compiler-matching) + - [Examples](#examples) + - [Invalid tool selections](#invalid-tool-selections) + - [CLI commands](#cli-commands) + - [Version syntax](#version-syntax) + - [Batch operations](#batch-operations) + - [Open question: version separator](#open-question-version-separator) + - [Open question: `add` vs `install` naming](#open-question-add-vs-install-naming) + - [Open question: `dune tools run` PATH fallback](#open-question-dune-tools-run-path-fallback) + - [Tool resolution](#tool-resolution) + - [Source resolution](#source-resolution) + - [Version selection](#version-selection) + - [Example scenarios](#example-scenarios) + - [Directory structure](#directory-structure) + - [Tool pforms](#tool-pforms) + - [Syntax](#syntax-1) + - [Resolution](#resolution) + - [Examples](#examples-1) + - [Future: library access](#future-library-access) +- [Relationship to Package Management](#relationship-to-package-management) + - [Orthogonality: a key design principle](#orthogonality-a-key-design-principle) + - [The package-tool continuum](#the-package-tool-continuum) + - [Key differences from project dependencies](#key-differences-from-project-dependencies) + - [Why tools don't use `dune.lock/`](#why-tools-dont-use-dunelock) + - [Locking vs building](#locking-vs-building) + - [Future: Relocatable compiler and caching](#future-relocatable-compiler-and-caching) + - [Potential future stanza options](#potential-future-stanza-options) +- [GitHub Issues Addressed](#github-issues-addressed) + - [Arbitrary tool packages ([#12913])](#arbitrary-tool-packages-12913) + - [Lock directory clutter ([#10955], [#12097])](#lock-directory-clutter-10955-12097) + - [Compiler ABI mismatch ([#11229])](#compiler-abi-mismatch-11229) + - [Graceful fallback for missing tools ([#10578])](#graceful-fallback-for-missing-tools-10578) + - [Version constraints ([#12777])](#version-constraints-12777) + - [Version consistency ([#10688])](#version-consistency-10688) + - [Dev tools redesign ([#12914])](#dev-tools-redesign-12914) + - [Tool dependency isolation ([#12551])](#tool-dependency-isolation-12551) + - [No project build triggers ([#11037])](#no-project-build-triggers-11037) + - [Compiler-independent formatters ([#11038])](#compiler-independent-formatters-11038) + - [Version from config files ([#5315])](#version-from-config-files-5315) + - [Version constraint parsing ([#12866])](#version-constraint-parsing-12866) + - [Opam with-dev-setup integration ([#12135])](#opam-with-dev-setup-integration-12135) + - [Helpful error messages ([#13235], [#12975])](#helpful-error-messages-13235-12975) + - [Batch installation ([#12557])](#batch-installation-12557) + - [Failure isolation ([#12818])](#failure-isolation-12818) + - [Utop findlib integration ([#13471])](#utop-findlib-integration-13471) +- [Comparison with Other Tools](#comparison-with-other-tools) + - [uv (Python)](#uv-python) + - [cargo (Rust)](#cargo-rust) + - [cargo-run-bin (Rust community tool)](#cargo-run-bin-rust-community-tool) + - [npm (JavaScript)](#npm-javascript) + - [How Our Design Compares](#how-our-design-compares) + - [Tools compared](#tools-compared) + - [Comparison table](#comparison-table) + - [Project-local](#project-local) + - [Multi-version](#multi-version) + - [Declarative config](#declarative-config) + - [Compiler matching](#compiler-matching-1) + - [PATH fallback](#path-fallback) + - [Precompiled binaries](#precompiled-binaries) + - [Tool isolation](#tool-isolation) + - [Ephemeral runs](#ephemeral-runs) + - [Batch install](#batch-install) + - [Binary selection](#binary-selection) + - [Tool discovery](#tool-discovery) + - [Tool upgrade](#tool-upgrade) + - [System-wide install](#system-wide-install) + - [Recommendations from Research](#recommendations-from-research) +- [Learnings from the Prototype](#learnings-from-the-prototype) + - [What Worked Well](#what-worked-well) + - [Problems Discovered](#problems-discovered) + - [Reference Packages for Testing](#reference-packages-for-testing) +- [Ideas](#ideas) -## How to Read This Document - -[Terminology](#terminology) defines key terms used throughout the document. -Precise definition of technical terms prevents misunderstandings and certain -classes of requirement error. - -[Design principles](#design-principles) specifies the guiding principles -informing the design. - -[Requirements](#requirements) defines _what_ capabilities the system must -provide and _why_. Organized by category. Requirements describe user-facing -behavior without specifying implementation details. Cross-references point to -the relevant Implementation sections. - -[Implementation](#implementation) details _how_ the system implements the requirements: -stanza syntax, CLI commands, version resolution algorithm, and directory -structure. Each section notes which requirements it implements. Open questions -are marked where decisions are pending. - -[Relationship to Package Management](#relationship-to-package-management) -explains the orthogonality principle and how tools differ from project -dependencies. - -[Comparison with Other Tools](comparison-with-other-tools) analyzes how uv, -cargo, cargo-run-bin, and npm handle tool management, informing our design -decisions. - -## Terminology - -- A **tool** is just an executable provided by some opam package. -- To **install** a tool is to make the executable available within the workspace. -- A **well formed opam package** specifies all data necessary to install and - build its provided targets. -- A tool is **installable** if it is part of a well formed opam package obtainable - from any source: opam repository, pinning from a source, or defined locally. - -## Design principles - -- **Generality**: Any executable defined in an opam packages can be a tool. -- **Orthogonality**: Tool management should not interfere with other systems, - including the management of other tools. - -## Requirements - -NOTE: The requirements here should be compatible with the functional -requirements in -[https://ocaml.org/tools/platform-roadmap](https://ocaml.org/tools/platform-roadmap). -However, wherever the roadmap specifies implementation details, we are free to -deviate if needed to better satisfy the requirements, and our task here is not -to realize all of the requirements in the platform roadmap. - -### 1. Installation - -Users must be able to install tools via Dune. - -#### 1.1. Generality - -Any installable tool must be supported. - -#### 1.2. Workspace-local - -Tools are installed per-workspace, not globally. Each workspace has its own -isolated tool installations that don't affect other workspaces or the system. - -See [Directory structure](#directory-structure) for storage locations. - -#### 1.3. System wide - -Users should be able to install tools in a way that allows them to be used in the -system-wide environment (e.g., outside of any particular sandbox). - -*NOTE:* This does not dictate that dune must maintain the equivalent of -default switches, or predetermine any other implementation choice. But the -support for tool management must be designed in way that makes it simple and -reliable for users to use installed tools outside of a project sandbox (e.g., by -adding the location of a directory of binaries to their `PATH` or some other -means). - -#### 1.4. Version specification - -Users must be able to install specific versions of tools via: - -- CLI arguments -- Declarative configuration -- Tool-specific configuration files (e.g., `.ocamlformat`) - -See [Version syntax](#version-syntax) for CLI syntax and -[The `(tool)` stanza](#the-tool-stanza) for declarative configuration. - -##### 1.4.1. Version consistency TODO - -What happens in case of conflict? - -- Stanza says `(= 0.26.2)` but CLI requests `0.27.0`: which wins? -- `.ocamlformat` says `version=0.26.2` but stanza says `(= 0.27.0)`: conflict? -- Version doesn't exist in opam-repository: error message? - -CR Shon: could we require consistency rather than needing to support precedence? -E.g., setting the version by CLI would cause an update to any config file, and -config files must be kept up to date? - -#### 1.5. Multi-version support - -Multiple versions of the same tool can coexist within a workspace. This enables -per-project tool resolution - for example, different projects within a workspace -can have different `.ocamlformat` files specifying different versions, and the -formatting rules will use the correct version for each project. - -See [Tool pforms](#tool-pforms) for how build rules can reference versioned tool -executables. - -#### 1.5. Clean source tree - -Tool lock directories and built artifacts must not pollute the source tree. They -should be stored in build output directories that are: - -- Excluded from version control -- Cleaned by standard build cleanup -- Invisible to users during normal development - -See [Directory structure](#directory-structure) for exact paths. - -CR-soon Alizter: This is a common complaint from users and is one such way to -solve the issue. Tools like uv handle this differently by having a global place. -Due to our compiler matching semantics it makes more sense for workspace level -and becomes fast with full caching. - -#### 1.6. Binary selection - -When a package provides multiple tools, users must be able to specify a subset -for installation. When a package providing tools is installed without -qualification, all provided tools must be installed. As a special case, when a -package provides a single binary, it will be installed without need to qualify. - -See [The `(tool)` stanza](#the-tool-stanza) and [CLI commands](#cli-commands) -for syntax. - -### 2. Usability - -Users must be able to run tools installed by Dune. - -#### 2.1. Shells - -Users must be able to run tools by invoking them directly in any shell (e.g., bash). - -See [CLI commands](#cli-commands) for invocation syntax. - -#### 2.2. Version specification TODO - -Users must be able run a specific version of a tool when multiple versions are installed. - -See [CLI commands](#cli-commands) for invocation syntax. - -CR-soon Alizter: Edge cases to specify: - -- Tool not locked: error with suggestion to run `dune tools add`? -- Multiple versions locked, none specified: error listing available versions? -- Specified version not locked: error with suggestion? -- Build fails: propagate build error? - -#### 2.3. Programmatic use - -Programs (e.g., editor plugins) must be able to find and run installed -tools via a single, transparent mechanism (e.g., the `dune tools env` equivalent -to `opam env` or by adding a single directory of executables to the lookup path, -or some other means). - -See [CLI commands](#cli-commands) for the discovery interface. - -CR-soon Alizter: `dune tools path` behavior is underspecified: - -- Should `path` trigger download and build if not yet built? Or only return path - if already built, erroring otherwise? -- If `--bin` is specified, can we compute the path without building (since path - is deterministic from package/version/binary name)? -- If package has multiple binaries and no `--bin`: error immediately, or - download and build first to discover available binaries? -- Tool not locked: error, or fall back to `which`? -- How does `path` interact with `(tool)` stanzas vs CLI-added tools? Same - resolution as `run`? - -CR-someday Alizter: Consider a single bin directory with symlinks to all tool -executables (like npm's `node_modules/.bin/`). Instead of per-tool paths, have -`_build/.tools/bin/` containing symlinks to all installed tool binaries. This -simplifies editor integration (one directory to add to PATH), avoids PATH length -limits (especially on Windows), and provides a single stable location for -discovery. Trade-off: need to maintain symlinks as tools are added/removed. - -##### 2.3.1 dune subcommands - -As a special case, dune subcommands that invoke external tools must be able to -use tools managed by `dune tools`. See [Dune Integration](#5-dune-integration). - -#### 2.5. System PATH fallback - -When a tool is not locked and no version is specified (e.g., `.ocamlformat` -without a version), dune subcommands should fall back to the system PATH. - -**Note** This is motivated by integration with editor developers who would like -a single point of truth for running tools, and for dune to handle it. This would -mean opam users can continue to use dune in which ever way they please and the -editors will not have to care. - -CR-Alizter soon: Fixup wording above. - -CR-Alizter soon: How would this even work if we don't know the binary from the -package? - -#### 2.6. Editor integration - -Editors and IDEs must be able to: - -- Discover which tools are available in a workspace -- Find the path to tool executables for spawning -- Invoke tools with dune managing the process (alternative pattern) - -See [CLI commands](#cli-commands) for the discovery and invocation interfaces. - -**Open issues** (require consultation with editors team): - -- How editors discover which tools are available -- LSP server invocation pattern: path discovery (editor spawns process) vs - managed invocation (dune manages process). LSP servers need editor control of - stdin/stdout for JSON-RPC and process lifecycle (restart on crash). -- Single-file formatting: `dune fmt ` for editor integration ([#3244] - discusses this) - -[#3244]: https://github.com/ocaml/dune/issues/3244 - -- `dune fmt` over RPC for watch mode integration -- Integration with `.ocamlformat` version detection -- Recommended editor configuration patterns - -CR-someday Alizter: Path staleness problem. If an editor caches the result of -`dune tools path ocamllsp` and the user later updates their OCaml version, the -cached path may point to an ABI-incompatible binary. There's no mechanism to -notify editors that cached paths are invalid. Options: (1) editors always call -`dune tools path` fresh, (2) dune provides a staleness check, (3) watch mode -integration notifies editors of tool rebuilds. - -### 3. Dependency interactions - -Tools installed by dune should have the minimal necessary interaction with other -dependencies of the environment they are installed in. - -#### 3.1. Compiler compatibility - -By default, tools are built with a compiler matching the environment. This -ensures tools like ocamllsp can read project build artifacts correctly. - -Users should be able to opt out of compiler matching per-tool for tools that -don't need it (e.g., formatters that only parse source text). - -CR-soon Alizter: The utility of opting out seems debatable, but is part of the -package-tool continuum approach. - -See [Compiler matching](#compiler-matching) for the detection algorithm. - -CR-someday Alizter: Reverse influence - could tools constrain the project? E.g., -if ocamllsp isn't available for OCaml 5.3 yet, a user might want their tool -requirements to influence which compiler version they use. Currently tools are -fully isolated and don't affect project solving. This would be a significant -departure from the orthogonality principle. - -#### 3.2. Dependency isolation - -Each tool is solved independently with its own lock directory. Tool dependencies -do not affect the project's `dune.lock`, and vice versa. - -See [Directory structure](#directory-structure) for lock directory locations. - -#### 3.3. No build triggers - -Locking or adding tools must not trigger project builds. - -Build rules that _use_ tools (e.g., formatting) are a separate concern; see -[Dune Integration](#5-dune-integration). - -#### 3.4. Watch mode integration - -Tool operations (`dune tools add`, `run`, etc.) must work correctly when a watch -server is running (`dune build -w`). Rather than directly manipulating lock -directories, tool commands should coordinate with the watch server via RPC to -avoid races and ensure the server picks up newly added tools. - -CR-soon Alizter: Specify the RPC protocol for tool operations. What messages are -needed? How does the watch server respond to tool additions? - -CR-soon Alizter: Document concurrent access behavior. What happens if two -terminals run `dune tools add` simultaneously? Or `add` while `run` is building? - -#### 3.5. Dog fooding - -Tools must work in the dune repository itself. Dune developers should be able to -run `dune tools add ocamlformat` and `dune tools add ocaml-lsp-server` when -working on dune, even though `dune pkg lock` doesn't work there due to the "in -and out" problem ([#8652]). - -[#8652]: https://github.com/ocaml/dune/issues/8652 - -This is enabled by the orthogonality design (3.2): tools are solved and built -independently from project dependencies, so they don't require a working project -lock directory. - -### 4. UI - -#### 4.1. CLI - -Users must be able to manage tools using CLI commands: - -- Add/lock individual tools to the workspace -- Run tools (building if needed) -- List locked tools and versions -- Remove tools -- Discover paths to tool executables - -CLI-added tools persist until `dune clean` or `dune tools remove`, but are not -reproducible across clean builds. See [Batch operations](#batch-operations) for -batch commands. - -#### 4.2. Persistent configuration - -Users must be able to declare tools in workspace configuration. Unlike CLI-added -tools, declared tools are reproducible: - -- Declaration survives `dune clean` (re-locked automatically from config) -- Declaration is version-controlled and shared with collaborators -- CI and fresh checkouts get the same tools - -See [The `(tool)` stanza](#the-tool-stanza) for syntax. - -### 5. Dune Integration - -Tools must integrate with existing dune features that rely on external -executables. - -#### 5.1. Format rules (`dune fmt`, `dune build @fmt`) - -`dune fmt` must use tools managed by `dune tools`. This includes respecting -version constraints from `.ocamlformat` files, PATH fallback -([2.4](#24-system-path-fallback)), and multi-version support -([1.4](#14-multi-version-support)). - -See [Tool resolution](#tool-resolution) for the resolution algorithm. - -#### 5.2. Documentation rules (`dune build @doc`, `dune ocaml doc`) - -`dune build @doc` must use tools managed by `dune tools`. When odoc is -configured as a tool, use the locked version; otherwise fall back to PATH -([2.4](#24-system-path-fallback)). - -See [Tool resolution](#tool-resolution) for the resolution algorithm. - -CR-soon Alizter: OCaml platform considerations - -#### 5.3. REPL (`dune utop`, `dune ocaml utop`) - -`dune utop` must use tools managed by `dune tools`. When utop is configured as -a tool, use the locked version; otherwise fall back to PATH -([2.4](#24-system-path-fallback)). - -See [Tool resolution](#tool-resolution) for the resolution algorithm. - -#### 5.4. Tool references in actions - -Build actions (both user-written rules and dune's internal rules like -formatting) must be able to reference tool executables via pforms. If the tool -is not locked, fall back to PATH (consistent with -[2.4](#24-system-path-fallback)). - -See [Tool pforms](#tool-pforms) in the Specification for syntax. - -#### 5.5. Legacy migration - -The legacy `.dev-tools.locks/` system is removed and replaced by this design. - -CR-soon Alizter: Detail the migration story: - -- Which CLI commands stay (with same or changed behavior)? -- Which CLI commands are removed? -- What happens to existing `.dev-tools.locks/` directories? -- User-facing migration guide (re-add tools via `dune tools add`) - -## Implementation - ### The `(tool)` stanza _Implements: [1.3 Version pinning](#13-version-pinning), @@ -678,6 +256,7 @@ dune tools add Lock all tools from (tool) st dune tools add [.] ... Lock specific tool(s) dune tools run [.] [--bin ] [-- ] Run a tool, passing arguments after -- +dune tools path Print path to the managed binary directory (or directories) dune tools path [.] [--bin ] Print path to tool executable dune tools list List all locked tools and versions dune tools remove [.] ... Remove a tool's lock directory @@ -1916,3 +1495,4 @@ separate solves, downloads, and builds. In a monorepo migrating ocamlformat versions across 50 sub-projects, users could accumulate many versions. Consider: (1) warning when version count exceeds threshold, (2) `dune tools gc` to remove unused versions, (3) documenting expected steady-state (few versions, not many). + diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md new file mode 100644 index 00000000000..ce4e8698983 --- /dev/null +++ b/doc/dev/tools-design/requirements.md @@ -0,0 +1,437 @@ +# Tools Requirements + +Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) + +## Summary + +This document specifies the requirements for dune's tool management system. + + + + +**Table of Contents** + +- [Tools Requirements](#tools-requirements) + - [Summary](#summary) + - [How to Read This Document](#how-to-read-this-document) + - [Terminology](#terminology) + - [Design principles](#design-principles) + - [Requirements](#requirements) + - [1. Installation](#1-installation) + - [1.1. Generality](#11-generality) + - [1.2. Workspace-local](#12-workspace-local) + - [1.3. System wide](#13-system-wide) + - [1.4. Version specification](#14-version-specification) + - [1.4.1. Version consistency TODO](#141-version-consistency-todo) + - [1.5. Multi-version support](#15-multi-version-support) + - [1.5. Clean source tree](#15-clean-source-tree) + - [1.6. Binary selection](#16-binary-selection) + - [2. Usability](#2-usability) + - [2.1. Shells](#21-shells) + - [2.2. Version specification TODO](#22-version-specification-todo) + - [2.3. Programmatic use](#23-programmatic-use) + - [2.3.1 dune subcommands](#231-dune-subcommands) + - [2.5. System PATH fallback](#25-system-path-fallback) + - [2.6. Editor integration](#26-editor-integration) + - [3. Dependency interactions](#3-dependency-interactions) + - [3.1. Compiler compatibility](#31-compiler-compatibility) + - [3.2. Dependency isolation](#32-dependency-isolation) + - [3.3. No build triggers](#33-no-build-triggers) + - [3.4. Watch mode integration](#34-watch-mode-integration) + - [3.5. Dog fooding](#35-dog-fooding) + - [4. UI](#4-ui) + - [4.1. CLI](#41-cli) + - [4.2. Persistent configuration](#42-persistent-configuration) + - [5. Dune Integration](#5-dune-integration) + - [5.1. Format rules (`dune fmt`, `dune build @fmt`)](#51-format-rules-dune-fmt-dune-build-fmt) + - [5.2. Documentation rules (`dune build @doc`, `dune ocaml doc`)](#52-documentation-rules-dune-build-doc-dune-ocaml-doc) + - [5.3. REPL (`dune utop`, `dune ocaml utop`)](#53-repl-dune-utop-dune-ocaml-utop) + - [5.4. Tool references in actions](#54-tool-references-in-actions) + - [5.5. Legacy migration](#55-legacy-migration) + + + +## How to Read This Document + +[Terminology](#terminology) defines key terms used throughout the document. +Precise definition of technical terms prevents misunderstandings and certain +classes of requirement error. + +[Design principles](#design-principles) specifies the guiding principles +informing the design. + +[Requirements](#requirements) defines _what_ capabilities the system must +provide and _why_. Organized by category. Requirements describe user-facing +behavior without specifying implementation details. Cross-references point to +the relevant Implementation sections. + +[Implementation](#implementation) details _how_ the system implements the requirements: +stanza syntax, CLI commands, version resolution algorithm, and directory +structure. Each section notes which requirements it implements. Open questions +are marked where decisions are pending. + +[Relationship to Package Management](#relationship-to-package-management) +explains the orthogonality principle and how tools differ from project +dependencies. + +[Comparison with Other Tools](comparison-with-other-tools) analyzes how uv, +cargo, cargo-run-bin, and npm handle tool management, informing our design +decisions. + +## Terminology + +- A **tool** is just an executable provided by some opam package. +- To **install** a tool is to make the executable available within the workspace. +- A **well formed opam package** specifies all data necessary to install and + build its provided targets. +- A tool is **installable** if it is part of a well formed opam package obtainable + from any source: opam repository, pinning from a source, or defined locally. + +## Design principles + +- **Generality**: Any executable defined in an opam packages can be a tool. +- **Orthogonality**: Tool management should not interfere with other systems, + including the management of other tools. + +## Requirements + +NOTE: The requirements here should be compatible with the functional +requirements in +[https://ocaml.org/tools/platform-roadmap](https://ocaml.org/tools/platform-roadmap). +However, wherever the roadmap specifies implementation details, we are free to +deviate if needed to better satisfy the requirements, and our task here is not +to realize all of the requirements in the platform roadmap. + +### 1. Installation + +Users must be able to install tools via Dune. + +#### 1.1. Generality + +Any installable tool must be supported. + +#### 1.2. Workspace-local + +Tools are installed per-workspace, not globally. Each workspace has its own +isolated tool installations that don't affect other workspaces or the system. + +See [Directory structure](./implementation.md#directory-structure) for storage locations. + +#### 1.3. System wide + +Users should be able to install tools in a way that allows them to be used in the +system-wide environment (e.g., outside of any particular sandbox). + +*NOTE:* This does not dictate that dune must maintain the equivalent of +default switches, or predetermine any other implementation choice. But the +support for tool management must be designed in way that makes it simple and +reliable for users to use installed tools outside of a project sandbox (e.g., by +adding the location of a directory of binaries to their `PATH` or some other +means). + +#### 1.4. Version specification + +Users must be able to install specific versions of tools via: + +- CLI arguments +- Declarative configuration +- Tool-specific configuration files (e.g., `.ocamlformat`) + +See [Version syntax](./implementation.md#version-syntax) for CLI syntax and +[The `(tool)` stanza](#the-tool-stanza) for declarative configuration. + +##### 1.4.1. Version consistency + +The versions of installed tools must remain consistent, accounting for all +configuration sources. E.g., consider an apparent conflict, such as a stanza +specifying `(= 0.26.2)` but a CLI input specifying `0.27.0`, `.ocamlformat` says +`version=0.26.2` but the stanza says `(= 0.27.0)`: in any such cases, a +consistent outcome must be derived. E.g., this could be achieved by having the +CLI input being used to update the config file, or simply by raising an error. +But it must not allow for a version to be installed that leads to inconsistent +version specifications. + +#### 1.5. Multi-version support + +Multiple versions of the same tool can coexist within a workspace. This enables +per-project tool resolution - for example, different projects within a workspace +can have different `.ocamlformat` files specifying different versions, and the +formatting rules will use the correct version for each project. + +See [Tool pforms](./implementation.md#tool-pforms) for how build rules can +reference versioned tool executables. + +#### 1.5. Clean source tree + +Tool lock directories and built artifacts must not pollute the source tree. They +should be stored in build output directories that are: + +- Excluded from version control +- Cleaned by standard build cleanup +- Invisible to users during normal development + +See [Directory structure](./implementation.md#directory-structure) for exact paths. + +**Motivation**: This is a common complaint from users and is one such way to +solve the issue. Tools like uv handle this differently by having a global place. +Due to our compiler matching semantics it makes more sense for workspace level +and becomes fast with full caching. + +#### 1.6. Binary selection + +When a package provides multiple tools, users must be able to specify a subset +for installation. When a package providing tools is installed without +qualification, all provided tools must be installed. As a special case, when a +package provides a single binary, it will be installed without need to qualify. + +See [The `(tool)` stanza](./implementation.md#the-tool-stanza) and [CLI +commands](./implementation.md#cli-commands) for syntax. + +### 2. Usability + +Users must be able to run tools installed by Dune. + +#### 2.1. Shells + +Users must be able to run tools by invoking them directly in any shell (e.g., bash). + +See [CLI commands](./implementation.md#cli-commands) for invocation syntax. + +#### 2.2. Version specification TODO + +Users must be able run a specific version of a tool when multiple versions are installed. + +See [CLI commands](./implementation.md#cli-commands) for invocation syntax. + +CR-soon Alizter: Edge cases to specify: + +- Tool not locked: error with suggestion to run `dune tools add`? +- Multiple versions locked, none specified: error listing available versions? +- Specified version not locked: error with suggestion? +- Build fails: propagate build error? + +#### 2.3. Programmatic use + +Programs (e.g., editor plugins) must be able to find and run installed +tools via a single, transparent mechanism (e.g., the `dune tools env` equivalent +to `opam env` or by adding a single directory of executables to the lookup path, +printing the path locations, or some other means). + +See [CLI commands](./implementation.md#cli-commands) for the discovery interface. + +CR-soon Alizter: `dune tools path` behavior is underspecified: + +- Should `path` trigger download and build if not yet built? Or only return path + if already built, erroring otherwise? +- If `--bin` is specified, can we compute the path without building (since path + is deterministic from package/version/binary name)? +- If package has multiple binaries and no `--bin`: error immediately, or + download and build first to discover available binaries? +- Tool not locked: error, or fall back to `which`? +- How does `path` interact with `(tool)` stanzas vs CLI-added tools? Same + resolution as `run`? + +CR-someday Alizter: Consider a single bin directory with symlinks to all tool +executables (like npm's `node_modules/.bin/`). Instead of per-tool paths, have +`_build/.tools/bin/` containing symlinks to all installed tool binaries. This +simplifies editor integration (one directory to add to PATH), avoids PATH length +limits (especially on Windows), and provides a single stable location for +discovery. Trade-off: need to maintain symlinks as tools are added/removed. + +##### 2.3.1 dune subcommands + +As a special case, dune subcommands that invoke external tools must be able to +use tools managed by `dune tools`. See [Dune Integration](#5-dune-integration). + +#### 2.5. System PATH fallback + +When a tool is not locked and no version is specified (e.g., `.ocamlformat` +without a version), dune subcommands should fall back to the system PATH. + +**Note** This is motivated by integration with editor developers who would like +a single point of truth for running tools, and for dune to handle it. This would +mean opam users can continue to use dune in which ever way they please and the +editors will not have to care. + +CR-Alizter soon: Fixup wording above. + +CR-Alizter soon: How would this even work if we don't know the binary from the +package? + +#### 2.6. Editor integration + +Editors and IDEs must be able to: + +- Discover which tools are available in a workspace +- Find the path to tool executables for spawning +- Invoke tools with dune managing the process (alternative pattern) + +See [CLI commands](./implementation.md#cli-commands) for the discovery and invocation interfaces. + +**Open issues** (require consultation with editors team): + +- How editors discover which tools are available +- LSP server invocation pattern: path discovery (editor spawns process) vs + managed invocation (dune manages process). LSP servers need editor control of + stdin/stdout for JSON-RPC and process lifecycle (restart on crash). +- Single-file formatting: `dune fmt ` for editor integration ([#3244] + discusses this) + +[#3244]: https://github.com/ocaml/dune/issues/3244 + +- `dune fmt` over RPC for watch mode integration +- Integration with `.ocamlformat` version detection +- Recommended editor configuration patterns + +CR-someday Alizter: Path staleness problem. If an editor caches the result of +`dune tools path ocamllsp` and the user later updates their OCaml version, the +cached path may point to an ABI-incompatible binary. There's no mechanism to +notify editors that cached paths are invalid. Options: (1) editors always call +`dune tools path` fresh, (2) dune provides a staleness check, (3) watch mode +integration notifies editors of tool rebuilds. + +### 3. Dependency interactions + +Tools installed by dune should have the minimal necessary interaction with other +dependencies of the environment they are installed in. + +#### 3.1. Compiler compatibility + +By default, tools are built with a compiler matching the environment. This +ensures tools like ocamllsp can read project build artifacts correctly. + +Users should be able to opt out of compiler matching per-tool for tools that +don't need it (e.g., formatters that only parse source text). + +CR-soon Alizter: The utility of opting out seems debatable, but is part of the +package-tool continuum approach. + +See [Compiler matching](./implementation.md#compiler-matching) for the detection +algorithm. + +CR-someday Alizter: Reverse influence - could tools constrain the project? E.g., +if ocamllsp isn't available for OCaml 5.3 yet, a user might want their tool +requirements to influence which compiler version they use. Currently tools are +fully isolated and don't affect project solving. This would be a significant +departure from the orthogonality principle. + +#### 3.2. Dependency isolation + +Each tool is solved independently with its own lock directory. Tool dependencies +do not affect the project's `dune.lock`, and vice versa. + +See [Directory structure](./implementation.md#directory-structure) for lock +directory locations. + +#### 3.3. No build triggers + +Locking or adding tools must not trigger project builds. + +Build rules that _use_ tools (e.g., formatting) are a separate concern; see +[Dune Integration](#5-dune-integration). + +#### 3.4. Watch mode integration + +Tool operations (`dune tools add`, `run`, etc.) must work correctly when a watch +server is running (`dune build -w`). Rather than directly manipulating lock +directories, tool commands should coordinate with the watch server via RPC to +avoid races and ensure the server picks up newly added tools. + +CR-soon Alizter: Specify the RPC protocol for tool operations. What messages are +needed? How does the watch server respond to tool additions? + +CR-soon Alizter: Document concurrent access behavior. What happens if two +terminals run `dune tools add` simultaneously? Or `add` while `run` is building? + +#### 3.5. Dog fooding + +Tools must work in the dune repository itself. Dune developers should be able to +run `dune tools add ocamlformat` and `dune tools add ocaml-lsp-server` when +working on dune. + +This is enabled by the orthogonality design (3.2): tools are solved and built +independently from project dependencies, so they don't require a working project +lock directory. + +### 4. UI + +#### 4.1. CLI + +Users must be able to manage tools using CLI commands: + +- Add/lock individual tools to the workspace +- Run tools (building if needed) +- List locked tools and versions +- Remove tools +- Discover paths to tool executables + +CLI-added tools persist until `dune clean` or `dune tools remove`, but are not +reproducible across clean builds. See [Batch +operations](./implementation.md#batch-operations) for batch commands. + +#### 4.2. Persistent configuration + +Users must be able to declare tools in workspace configuration. Unlike CLI-added +tools, declared tools are reproducible: + +- Declaration survives `dune clean` (re-locked automatically from config) +- Declaration is version-controlled and shared with collaborators +- CI and fresh checkouts get the same tools + +See [The `(tool)` stanza](./implementation.md#the-tool-stanza) for syntax. + +### 5. Dune Integration + +Tools must integrate with existing dune features that rely on external +executables. + +#### 5.1. Format rules (`dune fmt`, `dune build @fmt`) + +`dune fmt` must use tools managed by `dune tools`. This includes respecting +version constraints from `.ocamlformat` files, PATH fallback +([2.4](#24-system-path-fallback)), and multi-version support +([1.4](#14-multi-version-support)). + +See [Tool resolution](./implementation.md#tool-resolution) for the resolution algorithm. + +#### 5.2. Documentation rules (`dune build @doc`, `dune ocaml doc`) + +`dune build @doc` must use tools managed by `dune tools`. When odoc is +configured as a tool, use the locked version; otherwise fall back to PATH +([2.4](#24-system-path-fallback)). + +See [Tool resolution](./implementation.md#tool-resolution) for the resolution algorithm. + +CR-soon Alizter: OCaml platform considerations + +#### 5.3. REPL (`dune utop`, `dune ocaml utop`) + +`dune utop` must use tools managed by `dune tools`. When utop is configured as +a tool, use the locked version; otherwise fall back to PATH +([2.4](#24-system-path-fallback)). + +See [Tool resolution](./implementation.md#tool-resolution) for the resolution algorithm. + +#### 5.4. Tool references in actions + +Build actions (both user-written rules and dune's internal rules like +formatting) must be able to reference tool executables via pforms. If the tool +is not locked, fall back to PATH (consistent with +[2.4](#24-system-path-fallback)). + +See [Tool pforms](./implementation.md#tool-pforms) in the Specification for syntax. + +#### 5.5. Legacy migration + +The legacy `.dev-tools.locks/` system must be removed and replaced by this +design. Entailing updating all documentation and communicating needed users +interventions, as necessary. + + + + + + + From 8693e58b9c8eb2cd68d4334d9f7ec0440ff93581 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Wed, 8 Jul 2026 00:40:46 -0400 Subject: [PATCH 30/72] Update links and expand/tweak specs Signed-off-by: Shon Feder --- doc/dev/tools-design/implementation.md | 45 ++++++++++++++++---------- doc/dev/tools-design/requirements.md | 31 +++++++++--------- 2 files changed, 44 insertions(+), 32 deletions(-) diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md index 3087dbb0ecd..d155e0a26cd 100644 --- a/doc/dev/tools-design/implementation.md +++ b/doc/dev/tools-design/implementation.md @@ -95,10 +95,10 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni ### The `(tool)` stanza -_Implements: [1.3 Version pinning](#13-version-pinning), -[1.6 Binary selection](#16-binary-selection), -[3.1 Compiler compatibility](#31-compiler-compatibility), -[4.2 Persistent configuration](#42-persistent-configuration)_ +_Implements: [1.3 Version specification](./requirements.md#14-version-specification), +[1.6 Binary selection](./requirements.md#16-binary-selection), +[3.1 Compiler compatibility](./requirements.md#31-compiler-compatibility), +[4.2 Persistent configuration](./requirements.md#42-persistent-configuration)_ Tools are declared in `dune-workspace` using the `(tool)` stanza. The stanza uses the same dependency syntax as `(depends ...)` for the package field. @@ -155,7 +155,7 @@ CR-soon Alizter: Missing pins #### Compiler matching -_Implements: [3.1 Compiler compatibility](#31-compiler-compatibility)_ +_Implements: [3.1 Compiler compatibility](./requirements.md#31-compiler-compatibility)_ Tools that read project build artifacts (e.g., ocamllsp reading `.cmi` files) must be built with a compatible compiler to avoid ABI mismatches. By default, @@ -240,10 +240,9 @@ CR-soon Alizter: Specify exact error messages for these cases. ### CLI commands -_Implements: [1.3 Version pinning](#13-version-pinning), -[1.6 Binary selection](#16-binary-selection), -[2.1 Tool invocation](#21-tool-invocation), -[2.2 Programmatic use](#22-programmatic-use), [4.1 CLI](#41-cli)_ +_Implements: [1.4 Version specification](./requirements.md##14-version-specification), +[1.6 Binary selection](./requirements.md#16-binary-selection), +[2.3 Programmatic use](./requirements.md#23-programmatic-use), [4.1 CLI](./requirements.md#41-cli)_ CR-soon Alizter: The CLI commands are generally underspecified. Each command needs detailed behavior documentation: what triggers downloads/builds, how @@ -252,13 +251,17 @@ output format, etc. ``` dune tools List all locked tools (alias for list) +dune tools list List all locked tools and versions + dune tools add Lock all tools from (tool) stanzas dune tools add [.] ... Lock specific tool(s) + dune tools run [.] [--bin ] [-- ] Run a tool, passing arguments after -- + dune tools path Print path to the managed binary directory (or directories) dune tools path [.] [--bin ] Print path to tool executable -dune tools list List all locked tools and versions + dune tools remove [.] ... Remove a tool's lock directory ``` @@ -407,9 +410,9 @@ Considerations: ### Tool resolution -_Implements: [1.4 Multi-version support](#14-multi-version-support), -[2.4 System PATH fallback](#24-system-path-fallback), -[5.1 Format rules](#51-format-rules-dune-fmt)_ +_Implements: [1.5 Multi-version support](./requirements.md#15-multi-version-support), +[2.5 System PATH fallback](./requirements.md#25-system-path-fallback), +[5.1 Format rules](#51-format-rules)_ Tool resolution answers two questions: @@ -469,9 +472,9 @@ the locked version the most recent? ### Directory structure -_Implements: [1.2 Workspace-local](#12-workspace-local), -[1.5 Clean source tree](#15-clean-source-tree), -[3.2 Dependency isolation](#32-dependency-isolation)_ +_Implements: [1.2 Workspace-local](./requirements.md#12-workspace-local), +[1.5 Clean source tree](./requirements.md#15-clean-source-tree), +[3.2 Dependency isolation](./requirements.md#32-dependency-isolation)_ Tools use three types of directories: @@ -515,7 +518,15 @@ not inside `_private/`. This allows tools to be locked without going through the build system. The build lock directory and install directory are both inside `_private/default/` as they are managed by the build system. -### Tool pforms +### Tool pforms TODO + +CR-Shon: I don't understand why this is necessary: why can't actions just keep +using the `bin` pform? Introduction of yet another configuration quirk should +have very good motive, but we already have a way to refer to executables, and +tools are just a way to manage the installation of executables. + +Maybe what we actually want is just way of specifying the scope of tool +versions? Perhaps via an extension to `env`: this seems like an `env` issue... _Implements: [5.4 Tool references in actions](#54-tool-references-in-actions)_ diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index ce4e8698983..38ba47163d1 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -24,7 +24,7 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [1.2. Workspace-local](#12-workspace-local) - [1.3. System wide](#13-system-wide) - [1.4. Version specification](#14-version-specification) - - [1.4.1. Version consistency TODO](#141-version-consistency-todo) + - [1.4.1. Version consistency](#141-version-consistency) - [1.5. Multi-version support](#15-multi-version-support) - [1.5. Clean source tree](#15-clean-source-tree) - [1.6. Binary selection](#16-binary-selection) @@ -45,7 +45,7 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [4.1. CLI](#41-cli) - [4.2. Persistent configuration](#42-persistent-configuration) - [5. Dune Integration](#5-dune-integration) - - [5.1. Format rules (`dune fmt`, `dune build @fmt`)](#51-format-rules-dune-fmt-dune-build-fmt) + - [5.1. Format rules](#51-format-rules) - [5.2. Documentation rules (`dune build @doc`, `dune ocaml doc`)](#52-documentation-rules-dune-build-doc-dune-ocaml-doc) - [5.3. REPL (`dune utop`, `dune ocaml utop`)](#53-repl-dune-utop-dune-ocaml-utop) - [5.4. Tool references in actions](#54-tool-references-in-actions) @@ -205,12 +205,15 @@ Users must be able run a specific version of a tool when multiple versions are i See [CLI commands](./implementation.md#cli-commands) for invocation syntax. +CR Shon: how does, e.g., uv handle this? + CR-soon Alizter: Edge cases to specify: -- Tool not locked: error with suggestion to run `dune tools add`? -- Multiple versions locked, none specified: error listing available versions? -- Specified version not locked: error with suggestion? -- Build fails: propagate build error? +- Tool (or specified version of tool) not locked: + - error with suggestion to run `dune tools add`? +- Multiple versions locked, none specified: + - error listing available versions? + - default to newest version, or to a version marked as default? #### 2.3. Programmatic use @@ -387,12 +390,12 @@ See [The `(tool)` stanza](./implementation.md#the-tool-stanza) for syntax. Tools must integrate with existing dune features that rely on external executables. -#### 5.1. Format rules (`dune fmt`, `dune build @fmt`) +#### 5.1. Format rules -`dune fmt` must use tools managed by `dune tools`. This includes respecting -version constraints from `.ocamlformat` files, PATH fallback +`dune fmt` and `dune build @fmt` must use tools managed by `dune tools`. This +includes respecting version constraints from `.ocamlformat` files, PATH fallback ([2.4](#24-system-path-fallback)), and multi-version support -([1.4](#14-multi-version-support)). +([1.5](#15-multi-version-support)). See [Tool resolution](./implementation.md#tool-resolution) for the resolution algorithm. @@ -417,11 +420,9 @@ See [Tool resolution](./implementation.md#tool-resolution) for the resolution al #### 5.4. Tool references in actions Build actions (both user-written rules and dune's internal rules like -formatting) must be able to reference tool executables via pforms. If the tool -is not locked, fall back to PATH (consistent with -[2.4](#24-system-path-fallback)). - -See [Tool pforms](./implementation.md#tool-pforms) in the Specification for syntax. +formatting) must be able to reference the installed tools. E.g., via +`%{bin:...}` pforms, the base executable name, (or by some new mechanism, if +there is need for it). #### 5.5. Legacy migration From 7622a35ceb569c0e95795ddb5175ac019d7ad908 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 9 Jul 2026 23:26:18 -0400 Subject: [PATCH 31/72] Add Ambre and Sudha as authors Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 38ba47163d1..6c1ff45d57f 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -1,6 +1,7 @@ # Tools Requirements -Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) +Authors: Ali Caglayan (Tarides), Shon Feder (Tarides), Sudha Parimala (Tarides), +Ambre Suhamy (Tarides) ## Summary From 172fbfe0c08d403a05d260f5df8e960f14bcf31a Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 9 Jul 2026 23:26:53 -0400 Subject: [PATCH 32/72] Typo fixes Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 6c1ff45d57f..5a1dadbf0ed 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -65,21 +65,23 @@ informing the design. [Requirements](#requirements) defines _what_ capabilities the system must provide and _why_. Organized by category. Requirements describe user-facing -behavior without specifying implementation details. Cross-references point to +behaviour without specifying implementation details. Cross-references point to the relevant Implementation sections. -[Implementation](#implementation) details _how_ the system implements the requirements: + + +[Implementation](./implementation.md) details _how_ the system implements the requirements: stanza syntax, CLI commands, version resolution algorithm, and directory -structure. Each section notes which requirements it implements. Open questions +structure. Each section notes which requirements it implements. Open questions are marked where decisions are pending. -[Relationship to Package Management](#relationship-to-package-management) +[Relationship to Package Management](./implementation.md#relationship-to-package-management) explains the orthogonality principle and how tools differ from project dependencies. -[Comparison with Other Tools](comparison-with-other-tools) analyzes how uv, -cargo, cargo-run-bin, and npm handle tool management, informing our design -decisions. +[Comparison with Other Tools](./implementation.md#comparison-with-other-tools) +analyzes how `uv`,`cargo`, `cargo-run-bin`, and `npm` handle tool management, +informing our design decisions. ## Terminology From 2e2d909a5723d545773831caef07ddf5790e1d54 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 9 Jul 2026 23:27:05 -0400 Subject: [PATCH 33/72] Clarify terminology What is a workspace and the the one to many relationship between opam packages and dune tools. Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 5a1dadbf0ed..21a1e6381a0 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -85,12 +85,13 @@ informing our design decisions. ## Terminology -- A **tool** is just an executable provided by some opam package. -- To **install** a tool is to make the executable available within the workspace. -- A **well formed opam package** specifies all data necessary to install and +- A **tool** is just an executable provided by some opam package. One opam package can provide multiple tools. +- To **install** a tool is to make the executable available within an environment. +- A **well-formed opam package** specifies all data necessary to install and build its provided targets. -- A tool is **installable** if it is part of a well formed opam package obtainable +- A tool is **installable** if it is part of a well-formed opam package obtainable from any source: opam repository, pinning from a source, or defined locally. +- **Workspace** means a [dune workspace](https://dune.readthedocs.io/en/latest/explanation/scopes.html) ## Design principles From 8ca69035ee16f49770310467aa15b61a55b09d38 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 9 Jul 2026 23:40:02 -0400 Subject: [PATCH 34/72] Clarify common topic of "scope" Also introduce discussion of contexts Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 21a1e6381a0..96314c6e1f1 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -116,14 +116,29 @@ Users must be able to install tools via Dune. Any installable tool must be supported. -#### 1.2. Workspace-local +##### 1.2. Scope -Tools are installed per-workspace, not globally. Each workspace has its own -isolated tool installations that don't affect other workspaces or the system. +The environments within which a tool are available must be scoped. + +#### 1.2.1. Workspace-local scope + +It must be possible to install tools per-workspace, such that workspaces define +a sub-environment. I.e., each workspace has its own isolated tool installations +that don't affect other workspaces or interfere the system. See [Directory structure](./implementation.md#directory-structure) for storage locations. -#### 1.3. System wide +#### 1.2.2. Dune context scope + +CR Shon: Is this actually a good idea? How do tools interact with contexts? + +It must be possible to install the versions of a tool per-dune context within a +workspace, such that different contexts can use different versions of tools. + +This follows the example of `dune pkg`'s `lock_dir` in the context stanza, and +allows developers to set up different tooling configurations (What for?). + +#### 1.2.3. System wide scope Users should be able to install tools in a way that allows them to be used in the system-wide environment (e.g., outside of any particular sandbox). From f8144a0597c996d914cf5c11626a2d17ae664f04 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 9 Jul 2026 23:41:51 -0400 Subject: [PATCH 35/72] Clarify some wording Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 96314c6e1f1..fd0bfdea2a8 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -146,27 +146,27 @@ system-wide environment (e.g., outside of any particular sandbox). *NOTE:* This does not dictate that dune must maintain the equivalent of default switches, or predetermine any other implementation choice. But the support for tool management must be designed in way that makes it simple and -reliable for users to use installed tools outside of a project sandbox (e.g., by +reliable to use installed tools outside of a project sandbox (e.g., by adding the location of a directory of binaries to their `PATH` or some other means). #### 1.4. Version specification -Users must be able to install specific versions of tools via: +Users must be able to specify the version of tools to be installed via: - CLI arguments -- Declarative configuration +- Declarative configuration (i.e. dune stanzas) - Tool-specific configuration files (e.g., `.ocamlformat`) See [Version syntax](./implementation.md#version-syntax) for CLI syntax and -[The `(tool)` stanza](#the-tool-stanza) for declarative configuration. +[The `(tool)` stanza](./implementation.md#the-tool-stanza) for declarative configuration. ##### 1.4.1. Version consistency The versions of installed tools must remain consistent, accounting for all configuration sources. E.g., consider an apparent conflict, such as a stanza -specifying `(= 0.26.2)` but a CLI input specifying `0.27.0`, `.ocamlformat` says -`version=0.26.2` but the stanza says `(= 0.27.0)`: in any such cases, a +specifying `(= 0.26.2)` but a CLI input specifying `0.27.0`, or `.ocamlformat` says +`version=0.26.2` but the stanza says `(= 0.27.0)`: in such cases, a consistent outcome must be derived. E.g., this could be achieved by having the CLI input being used to update the config file, or simply by raising an error. But it must not allow for a version to be installed that leads to inconsistent From 4f4fbe195d949586f316d7e7256db329bd85eb44 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 9 Jul 2026 23:44:12 -0400 Subject: [PATCH 36/72] Remove requirements for multiple version installations - this seems over engineered, and only satisfies the one stated workflow (ocamlformat) (Sudha) - can we not require that tool versions be consistent within a workspace? People can use multiple workspaces if they need different versions (Sudha) - we already have a way to manage different versions and environments within the same workspace: contexts. If we want to be able to use multiple contexts within the same workspace side by side, perhaps we should address that at the level of contexts rather than tools. (Shon) - most other project managers don't allow this, but those that do (such as cargo or nix) support this on the level of packages per se, not just a subset. If we want to provide this kind of functionality, we should do it as a general solution on the level of package dependency specification, and pick it up from there, rather than special case it for tools. Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index fd0bfdea2a8..ccd48d1e461 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -172,15 +172,9 @@ CLI input being used to update the config file, or simply by raising an error. But it must not allow for a version to be installed that leads to inconsistent version specifications. -#### 1.5. Multi-version support -Multiple versions of the same tool can coexist within a workspace. This enables -per-project tool resolution - for example, different projects within a workspace -can have different `.ocamlformat` files specifying different versions, and the -formatting rules will use the correct version for each project. -See [Tool pforms](./implementation.md#tool-pforms) for how build rules can -reference versioned tool executables. + #### 1.5. Clean source tree From 93a684e8831694d9379004a2d5f5416e7aafb52c Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 9 Jul 2026 23:50:56 -0400 Subject: [PATCH 37/72] Remove implementation details clarify wording Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index ccd48d1e461..d6c9bb2a2cf 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -172,23 +172,14 @@ CLI input being used to update the config file, or simply by raising an error. But it must not allow for a version to be installed that leads to inconsistent version specifications. - - - - #### 1.5. Clean source tree -Tool lock directories and built artifacts must not pollute the source tree. They -should be stored in build output directories that are: - -- Excluded from version control -- Cleaned by standard build cleanup -- Invisible to users during normal development - -See [Directory structure](./implementation.md#directory-structure) for exact paths. +Tool lock directories and built artifacts must not pollute the source tree, to +ensure that they do not inadvertently picked up in version control or otherwise +create needless noise for users. **Motivation**: This is a common complaint from users and is one such way to -solve the issue. Tools like uv handle this differently by having a global place. +solve the issue. Tools like `uv` handle this differently by having a global place. Due to our compiler matching semantics it makes more sense for workspace level and becomes fast with full caching. From 06c9ce07956654d1cd2fb27cefceb16f9e98c0f8 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 9 Jul 2026 23:59:48 -0400 Subject: [PATCH 38/72] Add motivation for binary selection Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index d6c9bb2a2cf..251e6b4a285 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -188,11 +188,17 @@ and becomes fast with full caching. When a package provides multiple tools, users must be able to specify a subset for installation. When a package providing tools is installed without qualification, all provided tools must be installed. As a special case, when a -package provides a single binary, it will be installed without need to qualify. +package provides a single binary, it will be installed without needing to qualify. See [The `(tool)` stanza](./implementation.md#the-tool-stanza) and [CLI commands](./implementation.md#cli-commands) for syntax. +**Motivation**: This is necessitated by the fact that the relation between opam +packages and tools is one-to-many: a single package can provide multiple +executables. As a result, it becomes necessary to only install a preferred +subset of the provided tools. E.g., `js_of_ocaml-compiler` provides +`js_of_ocaml`, `jsoo_minify`, and `jsoo_listunits`. + ### 2. Usability Users must be able to run tools installed by Dune. From 36a43b27730dba84bcc705c4dfa6e35df44aa166 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 10 Jul 2026 00:00:06 -0400 Subject: [PATCH 39/72] Add requirement for installing tool by name Note: we now it this will take work beyond the scope of dune itself, it may not be a good candidate for the first version. Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 251e6b4a285..4904e6b1b59 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -199,6 +199,19 @@ executables. As a result, it becomes necessary to only install a preferred subset of the provided tools. E.g., `js_of_ocaml-compiler` provides `js_of_ocaml`, `jsoo_minify`, and `jsoo_listunits`. +#### 1.7 By tool name + +Users should be able to install tools based on the name of the tool without +considering the package that provides it. + +##### 1.7.1 Disambiguation + +If multiple packages provide the tools with the same name, and a user requests installation, dune should offer disambiguation. + +CR Shon: what do we do in dune package management if two packages provide the +same executable? E.g., perhaps you want to use package a for tool a' and b for +b', but they both also provide executables named `c`? + ### 2. Usability Users must be able to run tools installed by Dune. From 41d26ab991f210d0489063ad689be6cc829b77c6 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 10 Jul 2026 00:01:51 -0400 Subject: [PATCH 40/72] Add note on feasability Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 4904e6b1b59..122daa910ec 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -204,6 +204,10 @@ subset of the provided tools. E.g., `js_of_ocaml-compiler` provides Users should be able to install tools based on the name of the tool without considering the package that provides it. +CR Shon: because this will require changes to the opam repo to be effective, +this is a *should* rather than a *must* at the moment, and may not be achievable +in the first iteration of the redisign. + ##### 1.7.1 Disambiguation If multiple packages provide the tools with the same name, and a user requests installation, dune should offer disambiguation. From 77217e3aa4ccf19cc9034014ff379520aae72a72 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 10 Jul 2026 00:02:26 -0400 Subject: [PATCH 41/72] Remove req for running specific version of tool Matching removal of support for multiple versions Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 122daa910ec..c4712d83c5c 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -226,22 +226,6 @@ Users must be able to run tools by invoking them directly in any shell (e.g., ba See [CLI commands](./implementation.md#cli-commands) for invocation syntax. -#### 2.2. Version specification TODO - -Users must be able run a specific version of a tool when multiple versions are installed. - -See [CLI commands](./implementation.md#cli-commands) for invocation syntax. - -CR Shon: how does, e.g., uv handle this? - -CR-soon Alizter: Edge cases to specify: - -- Tool (or specified version of tool) not locked: - - error with suggestion to run `dune tools add`? -- Multiple versions locked, none specified: - - error listing available versions? - - default to newest version, or to a version marked as default? - #### 2.3. Programmatic use Programs (e.g., editor plugins) must be able to find and run installed From 47b5bd2fc4f013859d04339fac88492a0133e762 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 10 Jul 2026 00:05:16 -0400 Subject: [PATCH 42/72] Move implementation notes into correct document Signed-off-by: Shon Feder --- doc/dev/tools-design/implementation.md | 18 ++++++++++++++++++ doc/dev/tools-design/requirements.md | 19 ------------------- 2 files changed, 18 insertions(+), 19 deletions(-) diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md index d155e0a26cd..6596f1ac8ce 100644 --- a/doc/dev/tools-design/implementation.md +++ b/doc/dev/tools-design/implementation.md @@ -281,6 +281,24 @@ CR-soon Alizter: Edge cases for `dune tools add`: CR-soon Alizter: The prototype has `--allow-not-installed` flag for `dune tools path`. Document this flag and its use case (editor integration). +CR-soon Alizter: `dune tools path` behavior is underspecified: + +- Should `path` trigger download and build if not yet built? Or only return path + if already built, erroring otherwise? +- If `--bin` is specified, can we compute the path without building (since path + is deterministic from package/version/binary name)? +- If package has multiple binaries and no `--bin`: error immediately, or + download and build first to discover available binaries? +- Tool not locked: error, or fall back to `which`? +- How does `path` interact with `(tool)` stanzas vs CLI-added tools? Same + resolution as `run`? + +CR-someday Alizter: Consider a single bin directory with symlinks to all tool +executables (like npm's `node_modules/.bin/`). Instead of per-tool paths, have +`_build/.tools/bin/` containing symlinks to all installed tool binaries. This +simplifies editor integration (one directory to add to PATH), avoids PATH length +limits (especially on Windows), and provides a single stable location for +discovery. Trade-off: need to maintain symlinks as tools are added/removed. #### Version syntax The CLI uses dot-separated version syntax: `.` diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index c4712d83c5c..16c72a79541 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -235,25 +235,6 @@ printing the path locations, or some other means). See [CLI commands](./implementation.md#cli-commands) for the discovery interface. -CR-soon Alizter: `dune tools path` behavior is underspecified: - -- Should `path` trigger download and build if not yet built? Or only return path - if already built, erroring otherwise? -- If `--bin` is specified, can we compute the path without building (since path - is deterministic from package/version/binary name)? -- If package has multiple binaries and no `--bin`: error immediately, or - download and build first to discover available binaries? -- Tool not locked: error, or fall back to `which`? -- How does `path` interact with `(tool)` stanzas vs CLI-added tools? Same - resolution as `run`? - -CR-someday Alizter: Consider a single bin directory with symlinks to all tool -executables (like npm's `node_modules/.bin/`). Instead of per-tool paths, have -`_build/.tools/bin/` containing symlinks to all installed tool binaries. This -simplifies editor integration (one directory to add to PATH), avoids PATH length -limits (especially on Windows), and provides a single stable location for -discovery. Trade-off: need to maintain symlinks as tools are added/removed. - ##### 2.3.1 dune subcommands As a special case, dune subcommands that invoke external tools must be able to From 503213ec7087030070727a978943b5c9126fe8fa Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 10 Jul 2026 00:08:29 -0400 Subject: [PATCH 43/72] Fix req numbering Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 16c72a79541..a482af7a71d 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -226,7 +226,7 @@ Users must be able to run tools by invoking them directly in any shell (e.g., ba See [CLI commands](./implementation.md#cli-commands) for invocation syntax. -#### 2.3. Programmatic use +#### 2.2. Programmatic use Programs (e.g., editor plugins) must be able to find and run installed tools via a single, transparent mechanism (e.g., the `dune tools env` equivalent @@ -235,12 +235,12 @@ printing the path locations, or some other means). See [CLI commands](./implementation.md#cli-commands) for the discovery interface. -##### 2.3.1 dune subcommands +##### 2.2.1 dune subcommands As a special case, dune subcommands that invoke external tools must be able to use tools managed by `dune tools`. See [Dune Integration](#5-dune-integration). -#### 2.5. System PATH fallback +###### 2.2.1.1 System PATH fallback When a tool is not locked and no version is specified (e.g., `.ocamlformat` without a version), dune subcommands should fall back to the system PATH. @@ -255,7 +255,7 @@ CR-Alizter soon: Fixup wording above. CR-Alizter soon: How would this even work if we don't know the binary from the package? -#### 2.6. Editor integration +#### 2.3. Editor integration Editors and IDEs must be able to: From 8870bb97faad1393d6f228916fd550080c1ae9e7 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 10 Jul 2026 00:28:08 -0400 Subject: [PATCH 44/72] Move editor integration notes into implementation doc These are implementation details, and are covered on the requirements side by "programmatic use" Signed-off-by: Shon Feder --- doc/dev/tools-design/implementation.md | 32 ++++++++++++++++++++++ doc/dev/tools-design/requirements.md | 37 +------------------------- 2 files changed, 33 insertions(+), 36 deletions(-) diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md index 6596f1ac8ce..2eabd387c0e 100644 --- a/doc/dev/tools-design/implementation.md +++ b/doc/dev/tools-design/implementation.md @@ -1525,3 +1525,35 @@ versions across 50 sub-projects, users could accumulate many versions. Consider: (1) warning when version count exceeds threshold, (2) `dune tools gc` to remove unused versions, (3) documenting expected steady-state (few versions, not many). +### Editor integration + +Editors and IDEs must be able to: + +- Discover which tools are available in a workspace +- Find the path to tool executables for spawning +- Invoke tools with dune managing the process (alternative pattern) + +See [CLI commands](./implementation.md#cli-commands) for the discovery and invocation interfaces. + +**Open issues** (require consultation with editors team): + +- How editors discover which tools are available +- LSP server invocation pattern: path discovery (editor spawns process) vs + managed invocation (dune manages process). LSP servers need editor control of + stdin/stdout for JSON-RPC and process lifecycle (restart on crash). +- Single-file formatting: `dune fmt ` for editor integration ([#3244] + discusses this) + +[#3244]: https://github.com/ocaml/dune/issues/3244 + +- `dune fmt` over RPC for watch mode integration +- Integration with `.ocamlformat` version detection +- Recommended editor configuration patterns + +CR-someday Alizter: Path staleness problem. If an editor caches the result of +`dune tools path ocamllsp` and the user later updates their OCaml version, the +cached path may point to an ABI-incompatible binary. There's no mechanism to +notify editors that cached paths are invalid. Options: (1) editors always call +`dune tools path` fresh, (2) dune provides a staleness check, (3) watch mode +integration notifies editors of tool rebuilds. + diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index a482af7a71d..f2c04db8e2a 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -249,43 +249,8 @@ without a version), dune subcommands should fall back to the system PATH. a single point of truth for running tools, and for dune to handle it. This would mean opam users can continue to use dune in which ever way they please and the editors will not have to care. + -CR-Alizter soon: Fixup wording above. - -CR-Alizter soon: How would this even work if we don't know the binary from the -package? - -#### 2.3. Editor integration - -Editors and IDEs must be able to: - -- Discover which tools are available in a workspace -- Find the path to tool executables for spawning -- Invoke tools with dune managing the process (alternative pattern) - -See [CLI commands](./implementation.md#cli-commands) for the discovery and invocation interfaces. - -**Open issues** (require consultation with editors team): - -- How editors discover which tools are available -- LSP server invocation pattern: path discovery (editor spawns process) vs - managed invocation (dune manages process). LSP servers need editor control of - stdin/stdout for JSON-RPC and process lifecycle (restart on crash). -- Single-file formatting: `dune fmt ` for editor integration ([#3244] - discusses this) - -[#3244]: https://github.com/ocaml/dune/issues/3244 - -- `dune fmt` over RPC for watch mode integration -- Integration with `.ocamlformat` version detection -- Recommended editor configuration patterns - -CR-someday Alizter: Path staleness problem. If an editor caches the result of -`dune tools path ocamllsp` and the user later updates their OCaml version, the -cached path may point to an ABI-incompatible binary. There's no mechanism to -notify editors that cached paths are invalid. Options: (1) editors always call -`dune tools path` fresh, (2) dune provides a staleness check, (3) watch mode -integration notifies editors of tool rebuilds. ### 3. Dependency interactions From 2587de9c8132a70be520db6cf476e0e5f7871937 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 10 Jul 2026 00:29:11 -0400 Subject: [PATCH 45/72] Fix wording Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index f2c04db8e2a..baf36cd5476 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -228,22 +228,25 @@ See [CLI commands](./implementation.md#cli-commands) for invocation syntax. #### 2.2. Programmatic use -Programs (e.g., editor plugins) must be able to find and run installed -tools via a single, transparent mechanism (e.g., the `dune tools env` equivalent -to `opam env` or by adding a single directory of executables to the lookup path, -printing the path locations, or some other means). +Programs (e.g., editor plugins) must be able to find and run installed tools via +a single, transparent mechanism. (E.g., this could be implemented by any of the +following; the `dune tools env` equivalent to `opam env` or, `dune tools path` +that prints the paths to the binaries, by adding a single directory of +executables to the lookup path, printing the path locations, or some other +means). See [CLI commands](./implementation.md#cli-commands) for the discovery interface. ##### 2.2.1 dune subcommands -As a special case, dune subcommands that invoke external tools must be able to +As a special case, dune subcommands (e.g., `dune fmt` or `dune utop`) that invoke external tools must be able to use tools managed by `dune tools`. See [Dune Integration](#5-dune-integration). ###### 2.2.1.1 System PATH fallback -When a tool is not locked and no version is specified (e.g., `.ocamlformat` -without a version), dune subcommands should fall back to the system PATH. +When a tool is not installed (e.g., `.ocamlformat`), dune subcommands (such as +`dune fmt`, `dune build @doc`, or `dune utop`) should fall back to the system +PATH. **Note** This is motivated by integration with editor developers who would like a single point of truth for running tools, and for dune to handle it. This would From bee465a4f833c51e4f59d75d3fdfbfbb9534c02f Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 10 Jul 2026 00:29:22 -0400 Subject: [PATCH 46/72] Add orthoginal execuation req Entailed by principle Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index baf36cd5476..94ee5c38043 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -253,7 +253,10 @@ a single point of truth for running tools, and for dune to handle it. This would mean opam users can continue to use dune in which ever way they please and the editors will not have to care. +#### 2.4 Orthogonal execution +Running dune tools (by any means) should not interfere with other concurrent dune +operations (e.g., `dune build -w`). ### 3. Dependency interactions From 2d82fdd86131b969f22ae19eecf42fad53f15132 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 10 Jul 2026 00:31:40 -0400 Subject: [PATCH 47/72] Add notes on the tooling spectrum Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 89 ++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 94ee5c38043..de7e4e6865e 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -265,6 +265,9 @@ dependencies of the environment they are installed in. #### 3.1. Compiler compatibility +TODO: need to handle tools that are compiler integrations, and want to have an "optimal" dependency environment. +E.g., installing the latest version of a tool, with whatever compiler version it may need, or installing a tool that is not compatible with your env compiler. + By default, tools are built with a compiler matching the environment. This ensures tools like ocamllsp can read project build artifacts correctly. @@ -333,12 +336,16 @@ Users must be able to manage tools using CLI commands: - Remove tools - Discover paths to tool executables +TODO: we may want a req. that `dune clean` does not wipe out tools + CLI-added tools persist until `dune clean` or `dune tools remove`, but are not reproducible across clean builds. See [Batch operations](./implementation.md#batch-operations) for batch commands. #### 4.2. Persistent configuration +TODO: is "reproducible" too strong a word here? If no lock is checked in... + Users must be able to declare tools in workspace configuration. Unlike CLI-added tools, declared tools are reproducible: @@ -353,6 +360,8 @@ See [The `(tool)` stanza](./implementation.md#the-tool-stanza) for syntax. Tools must integrate with existing dune features that rely on external executables. +TODO: de duplicate following with usage notes on dune subcommands? + #### 5.1. Format rules `dune fmt` and `dune build @fmt` must use tools managed by `dune tools`. This @@ -382,6 +391,9 @@ See [Tool resolution](./implementation.md#tool-resolution) for the resolution al #### 5.4. Tool references in actions +CR Shon: anything referenced by an action is U2 or greater on the "integration +requirement spectrum". + Build actions (both user-written rules and dune's internal rules like formatting) must be able to reference the installed tools. E.g., via `%{bin:...}` pforms, the base executable name, (or by some new mechanism, if @@ -399,3 +411,80 @@ interventions, as necessary. + +# TODO (Shon, Ambre, Sudha): Analyze out the axes of use and integration + +This is about "the tool-dependency spectrum": most tools do not fit into the +binary between: + +- an adhoc binary a person wants to run a few times or +- a library dependency required for a project + +Most tools actually have fall on a spectrum with varying levels of integration +requirements, and various kinds of usage within the SDLC. + +Three ways of invoking a tool: + +- shell environment: (dune tools env) +- sub commands: dune fmt / dune build @doc +- dedicated dune runner subcommands: dune tools run/exec + +How do tools that are required for development currently interact with the + current deps? E.g., from opam, `(deps (odoc (and 2.1.1 :with-dev-setup)))` + +## There are at least 2 axes + +Each a spectrum, but we can sketch its shape using the two extremes and the +mixed case. + +### Usage categories spectrum: + +- U1. tools we want to install and run on an adhoc basis: dune tools install og; + og foo `dune build @tools` `dune build @editors-tools` :tools `dune pkg lock`. +- U2. tools we want to use as part of a build alias, but not as part of the + package dependencies (for tests, for build docs, fmt, could be geneneralized to + any build alias): with-doc, with-test, with-dev-setup +- U3. tools that are required for a build (e.g., menhir or atd), and just need + to be a package dependency unconditionally (but which users will also want to + be able to execute)! + +### Integration requirements spectrum: + +- I1. tools that must integrate with all other packages in a solution (e.g., + because of shared libraries) +- I2. tools that must integrate just with the compiler (aka "compiler + integrations" or "toolchain utilities") +- I3. tools that do not need to integrate with anything, and can be installed in + completely isolated environments. + +## Some example + +pkg foo + deps A.1 + +cmdliner --with-test + deps A.1 + +ocamllsp # exectable, but it hast to have library like integration (with compiler) + +ocamlformat + deps A.2 + +atd -> foo_j.ml (* as a fixuter json serde *) + deps A.3 + +## Some ideas + +Could consider a qualifier on executable packages + +- like "disjoint" or "sandboxed": but would have to be only for executables +- :exec-only, in which case (so long as not compiler integration!) can always be sandboxed + +Libs vs. executables. + +TODO: Consider use cases in many different forms and classify. + +- Menhir +- Ocsigen +- Coq/Fstar +- grep tools From abd92c3ad76200716742412428c9ba05bf6d76a0 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 10 Jul 2026 00:31:54 -0400 Subject: [PATCH 48/72] Add todo Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index de7e4e6865e..9eac4f99a66 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -401,6 +401,8 @@ there is need for it). #### 5.5. Legacy migration +TODO: is this just a task for our project, or is it actually a requirement? Is the requirement about backward compat. + The legacy `.dev-tools.locks/` system must be removed and replaced by this design. Entailing updating all documentation and communicating needed users interventions, as necessary. From d5c2bc0f997621781bb42efff3fa276888c3f891 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Sun, 12 Jul 2026 23:52:56 -0400 Subject: [PATCH 49/72] First stab at specifying the tool-dependency problem Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 49 +++++++++++++++++++++++++--- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 9eac4f99a66..ba4eaa1fbb2 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -1,4 +1,4 @@ -# Tools Requirements +# Tools Requirements like Authors: Ali Caglayan (Tarides), Shon Feder (Tarides), Sudha Parimala (Tarides), Ambre Suhamy (Tarides) @@ -85,8 +85,21 @@ informing our design decisions. ## Terminology -- A **tool** is just an executable provided by some opam package. One opam package can provide multiple tools. -- To **install** a tool is to make the executable available within an environment. +- A **tool** is just an executable provided by some opam package. One opam + package can provide multiple tools. +- A **project dependency** is a component (such as a library or executable) + required for developing and/or distributing a project. In the context of dune, + we know a component is a dependency if *any* of a project's build targets + depend upon it. +- A dependency is **qualified** when it is only required for a certain category + of build targets. Common qualification include *test* dependencies (e.g., + `alcotest`, `jq`, or `qcheck`) and *generation* dependencies (e.g., `atd`, + `menhir`, or `ocamlformat`). +- A tool is **discretionary**, relative to a particular project, if it is used + by some developers of the project, but it is not a *project dependency*. + Common examples include tools like `ocamllsp`, `utop`. +- To **install** a tool is to install the opam package providing it and making + the tool's executable available within an environment. - A **well-formed opam package** specifies all data necessary to install and build its provided targets. - A tool is **installable** if it is part of a well-formed opam package obtainable @@ -216,6 +229,27 @@ CR Shon: what do we do in dune package management if two packages provide the same executable? E.g., perhaps you want to use package a for tool a' and b for b', but they both also provide executables named `c`? +#### 1.8 Project dependency tools + +Tools that are *project dependencies* specified as appropriately qualified +dependencies in the `dune-project` file, must be installable via installation +targets reflecting the qualification. + +To illustrate, this could be thru some sort of qualification like `dune tools +install :with-test :with-dev-setup`. + +CR Shon: The need is legit here, but this requirement doesn't feel right, as is. +We need a general theory of how to think about qualified deps in dune managed +packages... + +##### 1.9 Discretionary tools + +It must be possible to install discretionary tools without incorrectly +specifying them as if they were project dependencies. + +#### 1.9 Discretionary tools + + ### 2. Usability Users must be able to run tools installed by Dune. @@ -255,8 +289,13 @@ editors will not have to care. #### 2.4 Orthogonal execution -Running dune tools (by any means) should not interfere with other concurrent dune -operations (e.g., `dune build -w`). +Running dune tools outside of a build (by any means) should not interfere with +other concurrent dune operations (e.g., `dune build -w`). + +#### 2.5 Project dependency tools + +When a tool is a *project dependency* (under any qualification), users must +be able to execute the tool. ### 3. Dependency interactions From 21819966b29ffacd75719f99bc636bf98248102a Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Tue, 14 Jul 2026 23:24:56 -0400 Subject: [PATCH 50/72] Updates from design discussions today Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 164 +++++++++++++++------------ 1 file changed, 89 insertions(+), 75 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index ba4eaa1fbb2..793927cc18f 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -233,22 +233,18 @@ b', but they both also provide executables named `c`? Tools that are *project dependencies* specified as appropriately qualified dependencies in the `dune-project` file, must be installable via installation -targets reflecting the qualification. +targets reflecting the qualification, as well as thru the build targets that +require them. -To illustrate, this could be thru some sort of qualification like `dune tools -install :with-test :with-dev-setup`. - -CR Shon: The need is legit here, but this requirement doesn't feel right, as is. -We need a general theory of how to think about qualified deps in dune managed -packages... +To illustrate, this could be thru some sort of qualification to the tools +command like `dune tools install :with-test :with-dev-setup`. ##### 1.9 Discretionary tools It must be possible to install discretionary tools without incorrectly -specifying them as if they were project dependencies. - -#### 1.9 Discretionary tools - +specifying them as if they were project dependencies. (E.g., thru a new `tool` +stanza or a CLI that updates some data stored in the workspace or `_build` +directory). ### 2. Usability @@ -273,8 +269,9 @@ See [CLI commands](./implementation.md#cli-commands) for the discovery interface ##### 2.2.1 dune subcommands -As a special case, dune subcommands (e.g., `dune fmt` or `dune utop`) that invoke external tools must be able to -use tools managed by `dune tools`. See [Dune Integration](#5-dune-integration). +As a special case, dune subcommands (e.g., `dune fmt` or `dune utop`) that +invoke external tools must be able to use tools managed by `dune tools`, with +fallback to executables available on the system path. ###### 2.2.1.1 System PATH fallback @@ -297,10 +294,22 @@ other concurrent dune operations (e.g., `dune build -w`). When a tool is a *project dependency* (under any qualification), users must be able to execute the tool. -### 3. Dependency interactions +### 3. Integration spectrum + +Tools lie along a spectrum of integration requirements with other dependencies +in the project, which we can indicate with these tree points: -Tools installed by dune should have the minimal necessary interaction with other -dependencies of the environment they are installed in. +- I1: At the minimum extreme, some tools can be solved and built in complete + isolation from the rest of the project they are used in. E.g., ocamlformat. +- I2: In the midpoint, some tools must integrate with a subset of a project's + dependencies. E.g., ocamllsp must use the same compiler version. +- I3: At the maximum extreme, some tools must be built within the entire + dependency context of the project. E.g., menhir when used together with its + runtime libraries, to ensure that the generated parser code is compatible with + the runtime library. + +Tools lying along the entire spectrum must be supported elegant solutions where +coupling is required, and the most possible orthogonality in features when it is not. #### 3.1. Compiler compatibility @@ -327,45 +336,56 @@ departure from the orthogonality principle. #### 3.2. Dependency isolation -Each tool is solved independently with its own lock directory. Tool dependencies -do not affect the project's `dune.lock`, and vice versa. +The dependencies of each tool should be solved independently with its own lock +directory. Tool dependencies do not affect the project's `dune.lock`, and vice +versa. See [Directory structure](./implementation.md#directory-structure) for lock directory locations. -#### 3.3. No build triggers -Locking or adding tools must not trigger project builds. +### 4. Dependency spectrum -Build rules that _use_ tools (e.g., formatting) are a separate concern; see -[Dune Integration](#5-dune-integration). +Tools lie along a spectrum of dependency status (i.e., to what extent they are +dependencies for the project) which we can indicate with these tree points: -#### 3.4. Watch mode integration +- D1. At the minimum extreme, *discretionary tools* are not project dependencies + at all. Rather, they are tools we want to install and run on an ad hoc basis, + for use as developers. E.g., `ocamllsp` or `ocamlgrep`. +- D2. In the midpoint, *qualified project dependency* tools are used in rules associated + with a build alias, but not as part of the required package dependencies to + install any packages. Example include any tools that are qualified in + opam packages with the `with-test`, or `with-doc` filters. +- D3. At the maximum extreme, unqualified *project dependency* tools are + required for an installation build of packages in the project, and need to be + specified as package dependencies unconditionally.E.g., `menhir` or `atd` when + used as part of the project build (by contrast, if these are used only for code + generation, they could be qualified project dependencies behind a hypothetical + `with-gen` filter). -Tool operations (`dune tools add`, `run`, etc.) must work correctly when a watch -server is running (`dune build -w`). Rather than directly manipulating lock -directories, tool commands should coordinate with the watch server via RPC to -avoid races and ensure the server picks up newly added tools. +#### 4.1. Discretionary tools (D1) -CR-soon Alizter: Specify the RPC protocol for tool operations. What messages are -needed? How does the watch server respond to tool additions? +##### 4.1.1. Not build triggers -CR-soon Alizter: Document concurrent access behavior. What happens if two -terminals run `dune tools add` simultaneously? Or `add` while `run` is building? +Locking or adding tools in at D1 level tools must not trigger project builds. -#### 3.5. Dog fooding +##### 4.1.2. Cannot be referenced in build rules TODO -Tools must work in the dune repository itself. Dune developers should be able to -run `dune tools add ocamlformat` and `dune tools add ocaml-lsp-server` when -working on dune. +#### 4.2. Qualified dependency tools (D2) TODO -This is enabled by the orthogonality design (3.2): tools are solved and built -independently from project dependencies, so they don't require a working project -lock directory. +##### 4.2.1. Must be usable as tools + +##### 4.2.1. Must be installable via qualification + +#### 4.3. Unqualified dependency tools (D3) TODO + +##### 4.3.1. Must be usable as tools TODO -### 4. UI +##### 4.2.1. Must always be installed TODO -#### 4.1. CLI +### 5. UI + +#### 5.1. CLI Users must be able to manage tools using CLI commands: @@ -381,7 +401,7 @@ CLI-added tools persist until `dune clean` or `dune tools remove`, but are not reproducible across clean builds. See [Batch operations](./implementation.md#batch-operations) for batch commands. -#### 4.2. Persistent configuration +#### 5.2. Persistent configuration TODO: is "reproducible" too strong a word here? If no lock is checked in... @@ -394,40 +414,6 @@ tools, declared tools are reproducible: See [The `(tool)` stanza](./implementation.md#the-tool-stanza) for syntax. -### 5. Dune Integration - -Tools must integrate with existing dune features that rely on external -executables. - -TODO: de duplicate following with usage notes on dune subcommands? - -#### 5.1. Format rules - -`dune fmt` and `dune build @fmt` must use tools managed by `dune tools`. This -includes respecting version constraints from `.ocamlformat` files, PATH fallback -([2.4](#24-system-path-fallback)), and multi-version support -([1.5](#15-multi-version-support)). - -See [Tool resolution](./implementation.md#tool-resolution) for the resolution algorithm. - -#### 5.2. Documentation rules (`dune build @doc`, `dune ocaml doc`) - -`dune build @doc` must use tools managed by `dune tools`. When odoc is -configured as a tool, use the locked version; otherwise fall back to PATH -([2.4](#24-system-path-fallback)). - -See [Tool resolution](./implementation.md#tool-resolution) for the resolution algorithm. - -CR-soon Alizter: OCaml platform considerations - -#### 5.3. REPL (`dune utop`, `dune ocaml utop`) - -`dune utop` must use tools managed by `dune tools`. When utop is configured as -a tool, use the locked version; otherwise fall back to PATH -([2.4](#24-system-path-fallback)). - -See [Tool resolution](./implementation.md#tool-resolution) for the resolution algorithm. - #### 5.4. Tool references in actions CR Shon: anything referenced by an action is U2 or greater on the "integration @@ -453,6 +439,34 @@ interventions, as necessary. + +# TODO Orphans to put in proper section + + +#### N. Watch mode integration + +Tool operations (e.g., `dune tools add`, `run`, etc.) must work correctly when a +watch server is running (`dune build -w`). Rather than directly manipulating +lock directories, tool commands should coordinate with the watch server via RPC +to avoid races and ensure the server picks up newly added tools. + +CR-soon Alizter: Specify the RPC protocol for tool operations. What messages are +needed? How does the watch server respond to tool additions? + +CR-soon Alizter: Document concurrent access behavior. What happens if two +terminals run `dune tools add` simultaneously? Or `add` while `run` is building? + +#### 4. Dog fooding + +Tools must work in the dune repository itself. Dune developers should be able to +run `dune tools add ocamlformat` and `dune tools add ocaml-lsp-server` when +working on dune. + +This is enabled by the orthogonality design (3.2): tools are solved and built +independently from project dependencies, so they don't require a working project +lock directory. + + # TODO (Shon, Ambre, Sudha): Analyze out the axes of use and integration This is about "the tool-dependency spectrum": most tools do not fit into the From 2df2031fa0fd98d74b9baa011ffe2f0edc00fcd2 Mon Sep 17 00:00:00 2001 From: Sudha Parimala Date: Thu, 16 Jul 2026 12:11:51 +0200 Subject: [PATCH 51/72] Add notes from issue triaging Relevant parts extracted from issues on the issue tracker tagged 'dev-tools' Signed-off-by: Sudha Parimala --- doc/dev/tools-design/implementation.md | 3 ++ doc/dev/tools-design/requirements.md | 65 +++++++++++++++++++++++++- 2 files changed, 67 insertions(+), 1 deletion(-) diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md index 2eabd387c0e..8c9d42d285e 100644 --- a/doc/dev/tools-design/implementation.md +++ b/doc/dev/tools-design/implementation.md @@ -126,6 +126,9 @@ CR-soon Alizter: Edge cases for stanza parsing: - Stanza references non-existent repository: when to error (parse time vs solve time)? +CR-soon Sudha247: Capture constraints here? Related issue: +[dune#12777](https://github.com/ocaml/dune/issues/12777) + #### Fields - **`(package )`** (required): The opam package providing the tool. Accepts diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 793927cc18f..2ee40a1fc5f 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -151,6 +151,14 @@ workspace, such that different contexts can use different versions of tools. This follows the example of `dune pkg`'s `lock_dir` in the context stanza, and allows developers to set up different tooling configurations (What for?). +
+ +Related issue: [dune#12557](https://github.com/ocaml/dune/issues/12557) + +#12557: dune tools install should take multiple package arguments + - Covered in the implentation spec. See (./implementation.md#batch-operations) +
+ #### 1.2.3. System wide scope Users should be able to install tools in a way that allows them to be used in the @@ -163,6 +171,15 @@ reliable to use installed tools outside of a project sandbox (e.g., by adding the location of a directory of binaries to their `PATH` or some other means). +
+ +Related issue: [dune#12107](https://github.com/ocaml/dune/issues/12107) + +#12107: pkg: installation of packages that can be used system-wide + - Even if we don't support this in the first version, the design should not + prevent this. +
+ #### 1.4. Version specification Users must be able to specify the version of tools to be installed via: @@ -239,6 +256,14 @@ require them. To illustrate, this could be thru some sort of qualification to the tools command like `dune tools install :with-test :with-dev-setup`. +
+ +Related issue: [dune#12135](https://github.com/ocaml/dune/issues/12135) + +#12135: dune tools setup to install :with-dev-setup deps + - Should the :with-dev-setup qualifier be used by dev-tools to install? +
+ ##### 1.9 Discretionary tools It must be possible to install discretionary tools without incorrectly @@ -246,6 +271,14 @@ specifying them as if they were project dependencies. (E.g., thru a new `tool` stanza or a CLI that updates some data stored in the workspace or `_build` directory). +
+ +Related issue: [dune#12913](https://github.com/ocaml/dune/issues/12913) + +#12913: pkg: general support for installing tools + - A tool can be any package with a binary. +
+ ### 2. Usability Users must be able to run tools installed by Dune. @@ -256,6 +289,20 @@ Users must be able to run tools by invoking them directly in any shell (e.g., ba See [CLI commands](./implementation.md#cli-commands) for invocation syntax. +
+ +Related issue: [dune#12975](https://github.com/ocaml/dune/issues/12975) + +#12975: running dune tools exec

when p is not already installed as a dev +tool should suggest users run dune tools install

+ - The error can be something like +``` +dune tools exec merlin + Error: The tool merlin is not installed + Hint: Try 'dune tools install merlin' +``` +

+ #### 2.2. Programmatic use Programs (e.g., editor plugins) must be able to find and run installed tools via @@ -283,7 +330,23 @@ PATH. a single point of truth for running tools, and for dune to handle it. This would mean opam users can continue to use dune in which ever way they please and the editors will not have to care. - + +CR Sudha247: do we need to split this into cases for when pkg is enabled and +disabled? If pkg is enabled, we don't provide system fallback as it could be +confusing to users if aliases just pickup binaries from stale OPAM switches. + +
+ +Related issue: [dune#10688](https://github.com/ocaml/dune/issues/10688) + +#10688: pkg: avoid dune fmt capturing ocamlformat from the PATH + - Install tools automatically? + - How would having two different package managers side by side work? (In this + case, Dune package and OPAM) + - Good to maintain compatibility with OPAM - but the tradeoff is the amount of + work needed to achieve it. +
+ #### 2.4 Orthogonal execution Running dune tools outside of a build (by any means) should not interfere with From dab2105dd083688ea3c191704d288489a5702d94 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 16 Jul 2026 09:24:10 -0400 Subject: [PATCH 52/72] Move impl stuff into impl doc Signed-off-by: Shon Feder --- doc/dev/tools-design/implementation.md | 21 ++++++++++++++++++++- doc/dev/tools-design/requirements.md | 15 --------------- 2 files changed, 20 insertions(+), 16 deletions(-) diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md index 8c9d42d285e..9258a90fbda 100644 --- a/doc/dev/tools-design/implementation.md +++ b/doc/dev/tools-design/implementation.md @@ -15,6 +15,7 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [Tools Implementation](#tools-implementation) - [Summary](#summary) +- [Design](#design) - [The `(tool)` stanza](#the-tool-stanza) - [Syntax](#syntax) - [Fields](#fields) @@ -32,7 +33,7 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [Version selection](#version-selection) - [Example scenarios](#example-scenarios) - [Directory structure](#directory-structure) - - [Tool pforms](#tool-pforms) + - [Tool pforms TODO](#tool-pforms-todo) - [Syntax](#syntax-1) - [Resolution](#resolution) - [Examples](#examples-1) @@ -90,9 +91,27 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [Problems Discovered](#problems-discovered) - [Reference Packages for Testing](#reference-packages-for-testing) - [Ideas](#ideas) + - [Editor integration](#editor-integration) +## How to Read This Document + +[Design](#design) details _how_ the system implements the requirements: +stanza syntax, CLI commands, version resolution algorithm, and directory +structure. Each section notes which requirements it implements. Open questions +are marked where decisions are pending. + +[Relationship to Package Management](#relationship-to-package-management) +explains the orthogonality principle and how tools differ from project +dependencies. + +[Comparison with Other Tools](#comparison-with-other-tools) analyzes how +`uv`,`cargo`, `cargo-run-bin`, and `npm` handle tool management, informing our +design decisions. + +## Design + ### The `(tool)` stanza _Implements: [1.3 Version specification](./requirements.md#14-version-specification), diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 2ee40a1fc5f..198762594f4 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -68,21 +68,6 @@ provide and _why_. Organized by category. Requirements describe user-facing behaviour without specifying implementation details. Cross-references point to the relevant Implementation sections. - - -[Implementation](./implementation.md) details _how_ the system implements the requirements: -stanza syntax, CLI commands, version resolution algorithm, and directory -structure. Each section notes which requirements it implements. Open questions -are marked where decisions are pending. - -[Relationship to Package Management](./implementation.md#relationship-to-package-management) -explains the orthogonality principle and how tools differ from project -dependencies. - -[Comparison with Other Tools](./implementation.md#comparison-with-other-tools) -analyzes how `uv`,`cargo`, `cargo-run-bin`, and `npm` handle tool management, -informing our design decisions. - ## Terminology - A **tool** is just an executable provided by some opam package. One opam From e979fbb61d0ef975068ce3a3bcafa696dd013228 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 16 Jul 2026 09:38:05 -0400 Subject: [PATCH 53/72] Clarify and extend principles Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 198762594f4..f294d2ff824 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -94,9 +94,14 @@ the relevant Implementation sections. ## Design principles - **Generality**: Any executable defined in an opam packages can be a tool. -- **Orthogonality**: Tool management should not interfere with other systems, - including the management of other tools. - +- **Orthogonality**: Dune's tool management functionality should compose with + other dune functionality with maximum flexibility and consistency, and without + introducing functional redundancies. +- **Complementarity**: Reciprocal to orthogonality, dune's tool management + functionality should integrate with and augment its core build functionality + and nascent package management functionality, and avoid introducing a bolted-on + subsystem. + ## Requirements NOTE: The requirements here should be compatible with the functional From c3903dabec200a496b18c03638238914051ee159 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 16 Jul 2026 09:41:10 -0400 Subject: [PATCH 54/72] Try to fix formatting Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index f294d2ff824..6ef78f017d5 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -99,8 +99,8 @@ the relevant Implementation sections. introducing functional redundancies. - **Complementarity**: Reciprocal to orthogonality, dune's tool management functionality should integrate with and augment its core build functionality - and nascent package management functionality, and avoid introducing a bolted-on - subsystem. + and nascent package management functionality, and avoid introducing a + bolted-on subsystem. ## Requirements @@ -143,7 +143,9 @@ allows developers to set up different tooling configurations (What for?).
+ Related issue: [dune#12557](https://github.com/ocaml/dune/issues/12557) + #12557: dune tools install should take multiple package arguments - Covered in the implentation spec. See (./implementation.md#batch-operations) From 317e5abb2ef731775a939140b7595a1f3b2be5ae Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 16 Jul 2026 09:45:45 -0400 Subject: [PATCH 55/72] Fix formatting Links aren't rendered in summary tags, and we will have additional details we want to add probably, so let's make the collapsed section more general. Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 6ef78f017d5..e00bc29829d 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -139,16 +139,19 @@ It must be possible to install the versions of a tool per-dune context within a workspace, such that different contexts can use different versions of tools. This follows the example of `dune pkg`'s `lock_dir` in the context stanza, and -allows developers to set up different tooling configurations (What for?). +allows developers to set up different tooling configurations (CR. What for?).
+Motivation and context + -Related issue: [dune#12557](https://github.com/ocaml/dune/issues/12557) +Related issues: + +- [dune#12557 dune tools install should take multiple package + arguments](https://github.com/ocaml/dune/issues/12557) + - Discussed in the implementation spec. See [./implementation.md#batch-operations](./implementation.md#batch-operations) - -#12557: dune tools install should take multiple package arguments - - Covered in the implentation spec. See (./implementation.md#batch-operations)
#### 1.2.3. System wide scope From 88e11c26d3403a3d6abdee884334b373aaf8a253 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 16 Jul 2026 09:51:58 -0400 Subject: [PATCH 56/72] Apply formatting fix to other locations Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 46 +++++++++++++++++++++------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index e00bc29829d..e042f80856b 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -168,11 +168,16 @@ means).
-Related issue: [dune#12107](https://github.com/ocaml/dune/issues/12107) +Motivation and context -#12107: pkg: installation of packages that can be used system-wide + +Related issues: + +- [dune#12107 pkg: installation of packages that can be used + system-wide](https://github.com/ocaml/dune/issues/12107) - Even if we don't support this in the first version, the design should not prevent this. +
#### 1.4. Version specification @@ -253,10 +258,15 @@ command like `dune tools install :with-test :with-dev-setup`.
-Related issue: [dune#12135](https://github.com/ocaml/dune/issues/12135) +Motivation and context -#12135: dune tools setup to install :with-dev-setup deps + +Related issues: + +- [dune#12135 dune tools setup to install :with-dev-setup + deps](https://github.com/ocaml/dune/issues/12135) - Should the :with-dev-setup qualifier be used by dev-tools to install? +
##### 1.9 Discretionary tools @@ -268,10 +278,15 @@ directory).
-Related issue: [dune#12913](https://github.com/ocaml/dune/issues/12913) +Motivation and context -#12913: pkg: general support for installing tools + +Related issues: + +- [dune#12913 pkg: general support for installing + tools](https://github.com/ocaml/dune/issues/12913) - A tool can be any package with a binary. +
### 2. Usability @@ -286,16 +301,20 @@ See [CLI commands](./implementation.md#cli-commands) for invocation syntax.
-Related issue: [dune#12975](https://github.com/ocaml/dune/issues/12975) +Motivation and context -#12975: running dune tools exec

when p is not already installed as a dev -tool should suggest users run dune tools install

+ +Related issues: + +- [dune#12975 running dune tools exec

when p is not already installed as a dev + tool should suggest users run dune tools install

](https://github.com/ocaml/dune/issues/12975) - The error can be something like ``` dune tools exec merlin Error: The tool merlin is not installed Hint: Try 'dune tools install merlin' ``` +

#### 2.2. Programmatic use @@ -332,14 +351,19 @@ confusing to users if aliases just pickup binaries from stale OPAM switches.
-Related issue: [dune#10688](https://github.com/ocaml/dune/issues/10688) +Motivation and context -#10688: pkg: avoid dune fmt capturing ocamlformat from the PATH + +Related issues: + +- [dune#10688 pkg: avoid dune fmt capturing ocamlformat from the + PATH](https://github.com/ocaml/dune/issues/10688) - Install tools automatically? - How would having two different package managers side by side work? (In this case, Dune package and OPAM) - Good to maintain compatibility with OPAM - but the tradeoff is the amount of work needed to achieve it. +
#### 2.4 Orthogonal execution From 7c55839e6ffc6da5042b9a17ec47684453751ac3 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 16 Jul 2026 09:52:32 -0400 Subject: [PATCH 57/72] Fix title Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index e042f80856b..3bea491badb 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -1,4 +1,4 @@ -# Tools Requirements like +# Dune Tools Requirements Authors: Ali Caglayan (Tarides), Shon Feder (Tarides), Sudha Parimala (Tarides), Ambre Suhamy (Tarides) From 744e40479b47580f1ae49191cc8994e108ebbad7 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Thu, 16 Jul 2026 10:03:53 -0400 Subject: [PATCH 58/72] Move issue to proper section It didn't make sense in the req it was under, and indeed we didn't have a req for it. Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 57 ++++++++++++++++++---------- 1 file changed, 38 insertions(+), 19 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 3bea491badb..1b98f2906ec 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -141,19 +141,6 @@ workspace, such that different contexts can use different versions of tools. This follows the example of `dune pkg`'s `lock_dir` in the context stanza, and allows developers to set up different tooling configurations (CR. What for?). -
- -Motivation and context - - -Related issues: - -- [dune#12557 dune tools install should take multiple package - arguments](https://github.com/ocaml/dune/issues/12557) - - Discussed in the implementation spec. See [./implementation.md#batch-operations](./implementation.md#batch-operations) - -
- #### 1.2.3. System wide scope Users should be able to install tools in a way that allows them to be used in the @@ -483,18 +470,50 @@ CLI-added tools persist until `dune clean` or `dune tools remove`, but are not reproducible across clean builds. See [Batch operations](./implementation.md#batch-operations) for batch commands. -#### 5.2. Persistent configuration +##### 5.1.1. Managing multiple tools -TODO: is "reproducible" too strong a word here? If no lock is checked in... +It must be possible to change (e.g., update, install, remove) the installation +of all configured tools by issuing a single command. -Users must be able to declare tools in workspace configuration. Unlike CLI-added -tools, declared tools are reproducible: +
+ +Motivation and context + + +We must not require users to do the tedious work of running the same command +over and over to install a set of tools. How this is addressed is left as an +implementation detail. E.g., it could mean supporting `dune tools install a b c` +or just having a command that installs all configured tools (see 5.2) in one +command. + +Related issues: + +- [dune#12557 dune tools install should take multiple package + arguments](https://github.com/ocaml/dune/issues/12557) + - Discussed in the implementation spec. See [./implementation.md#batch-operations](./implementation.md#batch-operations) + +
+ + +#### 5.2. Persistent configuration for discretionary tools + +Users must be able to declare discretionary tools in the workspace +configuration. + +
+ +Motivation and context + + +This allows users to share tooling configurations between users and across fresh +project setups as part of a reusable ad hoc dev tool setup. -- Declaration survives `dune clean` (re-locked automatically from config) - Declaration is version-controlled and shared with collaborators - CI and fresh checkouts get the same tools -See [The `(tool)` stanza](./implementation.md#the-tool-stanza) for syntax. +See [The `(tool)` stanza](./implementation.md#the-tool-stanza) for proposed syntax. + +
#### 5.4. Tool references in actions From c9786d4e8ad8c139c3f42856021b931ac1cb1171 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 17 Jul 2026 00:00:37 -0400 Subject: [PATCH 59/72] Update after our design meeting today Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 122 ++++++++++++++++++--------- 1 file changed, 83 insertions(+), 39 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 1b98f2906ec..e55873eed0a 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -73,9 +73,10 @@ the relevant Implementation sections. - A **tool** is just an executable provided by some opam package. One opam package can provide multiple tools. - A **project dependency** is a component (such as a library or executable) - required for developing and/or distributing a project. In the context of dune, - we know a component is a dependency if *any* of a project's build targets - depend upon it. + required for developing and/or distributing a project. For dune, this + generally means the component must be provided by a package dependency of some + package in the project. A component is a dependency if *any* of a project's + build targets depend upon it, whether or not it is correctly declared as such. - A dependency is **qualified** when it is only required for a certain category of build targets. Common qualification include *test* dependencies (e.g., `alcotest`, `jq`, or `qcheck`) and *generation* dependencies (e.g., `atd`, @@ -84,11 +85,13 @@ the relevant Implementation sections. by some developers of the project, but it is not a *project dependency*. Common examples include tools like `ocamllsp`, `utop`. - To **install** a tool is to install the opam package providing it and making - the tool's executable available within an environment. + the tool's executable available within an environment. The system responsible + for installing and uninstalling a tool is said to **manage** the tool. - A **well-formed opam package** specifies all data necessary to install and build its provided targets. -- A tool is **installable** if it is part of a well-formed opam package obtainable - from any source: opam repository, pinning from a source, or defined locally. +- A tool is **installable** if it is part of a well-formed opam package + obtainable from any source: opam repository, pinning from a source, or defined + locally. - **Workspace** means a [dune workspace](https://dune.readthedocs.io/en/latest/explanation/scopes.html) ## Design principles @@ -280,7 +283,32 @@ Related issues: Users must be able to run tools installed by Dune. -#### 2.1. Shells +#### 2.1 Extent of tool management + +The extent to which dune enforces management of tools within a workspace should +depend on whether or not package management is enabled in the workspace. + +##### 2.1.1 when package management is enabled in a workspace + +When users enable dune package management in a workspace, all *tools* used in the +workspace (in the precise sense defined in the [terminology](#terminology)) +should be managed dune, to the extent that dune can reasonably enforce this. +Enforcing this assumption allows dune to offer users improved guarantees about +the cohesiveness and interoperability of the provided tools. + +This does *not* entail that dune should mask or redact data from the `PATH` or +otherwise attempt to filter out a users ambient environment. But it should +provide pragmatic measures to support users by enforcing this behavior where +feasible. + +##### 2.1.2 when package management is not enabled in a workspace + +When users have not enabled dune package management in a workspace, they must be +able to use *tools* managed dune, but they should still be able to use tools +installed by opam (or other possible package managers) in all operations of +dune. + +#### 2.2. Shells Users must be able to run tools by invoking them directly in any shell (e.g., bash). @@ -304,7 +332,7 @@ dune tools exec merlin
-#### 2.2. Programmatic use +#### 2.3. Programmatic use Programs (e.g., editor plugins) must be able to find and run installed tools via a single, transparent mechanism. (E.g., this could be implemented by any of the @@ -315,27 +343,37 @@ means). See [CLI commands](./implementation.md#cli-commands) for the discovery interface. -##### 2.2.1 dune subcommands +##### 2.3.1. dune subcommands As a special case, dune subcommands (e.g., `dune fmt` or `dune utop`) that -invoke external tools must be able to use tools managed by `dune tools`, with -fallback to executables available on the system path. +invoke external tools must be able to use tools managed by `dune tools`, when +they are available. -###### 2.2.1.1 System PATH fallback +###### 2.3.1.1. System PATH fallback -When a tool is not installed (e.g., `.ocamlformat`), dune subcommands (such as -`dune fmt`, `dune build @doc`, or `dune utop`) should fall back to the system -PATH. +The extent to which dune should allow its subcommands to fallback to the system +`PATH` when looking up required binaries depends on whether or not the workspace +has enabled package management, as dictated by +[2.1](#2-1-extent-of-tool-management). + +###### 2.3.1.1.1. When package management is enabled in a workspace + +When package management is enabled in a workspace but a required tool is not +installed, dune subcommands must produce a clear user error explaining to +users that the tool is not available and directing them to install it as a dune +managed tool. + +###### 2.3.1.1.2. When package management is NOT enabled in a workspace + +When package management is enabled in a workspace and a tool is not installed +(e.g., `.ocamlformat`), dune subcommands (such as `dune fmt`, `dune build @doc`, +or `dune utop`) should fall back to the system PATH. **Note** This is motivated by integration with editor developers who would like a single point of truth for running tools, and for dune to handle it. This would mean opam users can continue to use dune in which ever way they please and the editors will not have to care. -CR Sudha247: do we need to split this into cases for when pkg is enabled and -disabled? If pkg is enabled, we don't provide system fallback as it could be -confusing to users if aliases just pickup binaries from stale OPAM switches. -
Motivation and context @@ -353,17 +391,23 @@ Related issues:
-#### 2.4 Orthogonal execution +#### 2.4. Project dependency tools + +When a tool is installed as a *project dependency* (under any qualification), +users must be able to execute the tool thru all the same mechanism that are +provided for executing discretionary tools. E.g., a mechanism like `dune env` +that would make the path to dune-managed binaries available must include both +discretionary tools and project dependency tools in the path. -Running dune tools outside of a build (by any means) should not interfere with -other concurrent dune operations (e.g., `dune build -w`). +### 3. Dependency and Integration -#### 2.5 Project dependency tools +Tools are used in numerous ways and require different levels of integration and +interdependence with the other components (tools, libraries, or other artifacts) +of a workspace. We can position these varieties of difference on a coordinate +system with two axes: -When a tool is a *project dependency* (under any qualification), users must -be able to execute the tool. -### 3. Integration spectrum +#### 3.1 Integration spectrum Tools lie along a spectrum of integration requirements with other dependencies in the project, which we can indicate with these tree points: @@ -380,7 +424,7 @@ in the project, which we can indicate with these tree points: Tools lying along the entire spectrum must be supported elegant solutions where coupling is required, and the most possible orthogonality in features when it is not. -#### 3.1. Compiler compatibility +##### 3.1. Compiler compatibility TODO: need to handle tools that are compiler integrations, and want to have an "optimal" dependency environment. E.g., installing the latest version of a tool, with whatever compiler version it may need, or installing a tool that is not compatible with your env compiler. @@ -403,7 +447,7 @@ requirements to influence which compiler version they use. Currently tools are fully isolated and don't affect project solving. This would be a significant departure from the orthogonality principle. -#### 3.2. Dependency isolation +##### 3.2. Dependency isolation The dependencies of each tool should be solved independently with its own lock directory. Tool dependencies do not affect the project's `dune.lock`, and vice @@ -413,7 +457,7 @@ See [Directory structure](./implementation.md#directory-structure) for lock directory locations. -### 4. Dependency spectrum +#### 4.2 Dependency spectrum Tools lie along a spectrum of dependency status (i.e., to what extent they are dependencies for the project) which we can indicate with these tree points: @@ -432,27 +476,27 @@ dependencies for the project) which we can indicate with these tree points: generation, they could be qualified project dependencies behind a hypothetical `with-gen` filter). -#### 4.1. Discretionary tools (D1) +##### 4.1. Discretionary tools (D1) -##### 4.1.1. Not build triggers +###### 4.1.1. Not build triggers Locking or adding tools in at D1 level tools must not trigger project builds. -##### 4.1.2. Cannot be referenced in build rules TODO +###### 4.1.2. Cannot be referenced in build rules TODO -#### 4.2. Qualified dependency tools (D2) TODO +##### 4.2. Qualified dependency tools (D2) TODO -##### 4.2.1. Must be usable as tools +###### 4.2.1. Must be usable as tools -##### 4.2.1. Must be installable via qualification +###### 4.2.1. Must be installable via qualification -#### 4.3. Unqualified dependency tools (D3) TODO +##### 4.3. Unqualified dependency tools (D3) TODO -##### 4.3.1. Must be usable as tools TODO +###### 4.3.1. Must be usable as tools TODO -##### 4.2.1. Must always be installed TODO +###### 4.2.1. Must always be installed TODO -### 5. UI +#### 5. UI #### 5.1. CLI From c4346d7b3294718b0e77ab42968d05d58ba229f4 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 17 Jul 2026 19:51:18 -0400 Subject: [PATCH 60/72] Update TOC Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 75 ++++++++++++++++++---------- 1 file changed, 48 insertions(+), 27 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index e55873eed0a..f9793d044f5 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -14,7 +14,7 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni **Table of Contents** -- [Tools Requirements](#tools-requirements) +- [Tools Requirements like](#tools-requirements-like) - [Summary](#summary) - [How to Read This Document](#how-to-read-this-document) - [Terminology](#terminology) @@ -22,35 +22,57 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [Requirements](#requirements) - [1. Installation](#1-installation) - [1.1. Generality](#11-generality) - - [1.2. Workspace-local](#12-workspace-local) - - [1.3. System wide](#13-system-wide) + - [1.2. Scope](#12-scope) + - [1.2.1. Workspace-local scope](#121-workspace-local-scope) + - [1.2.2. Dune context scope](#122-dune-context-scope) + - [1.2.3. System wide scope](#123-system-wide-scope) - [1.4. Version specification](#14-version-specification) - [1.4.1. Version consistency](#141-version-consistency) - - [1.5. Multi-version support](#15-multi-version-support) - [1.5. Clean source tree](#15-clean-source-tree) - [1.6. Binary selection](#16-binary-selection) + - [1.7 By tool name](#17-by-tool-name) + - [1.7.1 Disambiguation](#171-disambiguation) + - [1.8 Project dependency tools](#18-project-dependency-tools) + - [1.9 Discretionary tools](#19-discretionary-tools) - [2. Usability](#2-usability) - - [2.1. Shells](#21-shells) - - [2.2. Version specification TODO](#22-version-specification-todo) + - [2.1 Extent of tool management ](#21-extent-of-tool-management) + - [2.1.1 when package management is enabled in a workspace](#211-when-package-management-is-enabled-in-a-workspace) + - [2.1.2 when package management is not enabled in a workspace](#212-when-package-management-is-not-enabled-in-a-workspace) + - [2.2. Shells](#22-shells) - [2.3. Programmatic use](#23-programmatic-use) - - [2.3.1 dune subcommands](#231-dune-subcommands) - - [2.5. System PATH fallback](#25-system-path-fallback) - - [2.6. Editor integration](#26-editor-integration) - - [3. Dependency interactions](#3-dependency-interactions) - - [3.1. Compiler compatibility](#31-compiler-compatibility) - - [3.2. Dependency isolation](#32-dependency-isolation) - - [3.3. No build triggers](#33-no-build-triggers) - - [3.4. Watch mode integration](#34-watch-mode-integration) - - [3.5. Dog fooding](#35-dog-fooding) - - [4. UI](#4-ui) - - [4.1. CLI](#41-cli) - - [4.2. Persistent configuration](#42-persistent-configuration) - - [5. Dune Integration](#5-dune-integration) - - [5.1. Format rules](#51-format-rules) - - [5.2. Documentation rules (`dune build @doc`, `dune ocaml doc`)](#52-documentation-rules-dune-build-doc-dune-ocaml-doc) - - [5.3. REPL (`dune utop`, `dune ocaml utop`)](#53-repl-dune-utop-dune-ocaml-utop) + - [2.3.1. dune subcommands](#231-dune-subcommands) + - [2.3.1.1. System PATH fallback](#2311-system-path-fallback) + - [2.3.1.1.1. When package management is enabled in a workspace](#23111-when-package-management-is-enabled-in-a-workspace) + - [2.3.1.1.2. When package management is NOT enabled in a workspace ](#23112-when-package-management-is-not-enabled-in-a-workspace) + - [2.4. Project dependency tools](#24-project-dependency-tools) + - [3. Dependency and Integration](#3-dependency-and-integration) + - [3.1 Integration spectrum](#31-integration-spectrum) + - [3.1. Compiler compatibility](#31-compiler-compatibility) + - [3.2. Dependency isolation](#32-dependency-isolation) + - [4.2 Dependency spectrum](#42-dependency-spectrum) + - [4.1. Discretionary tools (D1)](#41-discretionary-tools-d1) + - [4.1.1. Not build triggers](#411-not-build-triggers) + - [4.1.2. Cannot be referenced in build rules TODO](#412-cannot-be-referenced-in-build-rules-todo) + - [4.2. Qualified dependency tools (D2) TODO](#42-qualified-dependency-tools-d2-todo) + - [4.2.1. Must be usable as tools](#421-must-be-usable-as-tools) + - [4.2.1. Must be installable via qualification](#421-must-be-installable-via-qualification) + - [4.3. Unqualified dependency tools (D3) TODO](#43-unqualified-dependency-tools-d3-todo) + - [4.3.1. Must be usable as tools TODO](#431-must-be-usable-as-tools-todo) + - [4.2.1. Must always be installed TODO](#421-must-always-be-installed-todo) + - [5. UI](#5-ui) + - [5.1. CLI](#51-cli) + - [5.2. Persistent configuration](#52-persistent-configuration) - [5.4. Tool references in actions](#54-tool-references-in-actions) - [5.5. Legacy migration](#55-legacy-migration) +- [TODO Orphans to put in proper section](#todo-orphans-to-put-in-proper-section) + - [-](#-) + - [4. Dog fooding](#4-dog-fooding) +- [TODO (Shon, Ambre, Sudha): Analyze out the axes of use and integration](#todo-shon-ambre-sudha-analyze-out-the-axes-of-use-and-integration) + - [There are at least 2 axes](#there-are-at-least-2-axes) + - [Usage categories spectrum:](#usage-categories-spectrum) + - [Integration requirements spectrum:](#integration-requirements-spectrum) + - [Some example](#some-example) + - [Some ideas](#some-ideas) @@ -401,11 +423,10 @@ discretionary tools and project dependency tools in the path. ### 3. Dependency and Integration -Tools are used in numerous ways and require different levels of integration and -interdependence with the other components (tools, libraries, or other artifacts) -of a workspace. We can position these varieties of difference on a coordinate -system with two axes: - +Tools are used in numerous ways in projects and for ad hoc developer needs, and +they require different levels of integration and interdependence with the other +components (tools, libraries, or other artifacts) of a workspace. We can +position these varieties of difference on a coordinate system with two axes: #### 3.1 Integration spectrum From 7cec859378dbc8bd42c50717e48cb4b5b4fe174c Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Fri, 17 Jul 2026 22:13:32 -0400 Subject: [PATCH 61/72] Fix related issues location Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 32 ++++++++++++++-------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index f9793d044f5..f71f9736bde 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -171,11 +171,11 @@ allows developers to set up different tooling configurations (CR. What for?). Users should be able to install tools in a way that allows them to be used in the system-wide environment (e.g., outside of any particular sandbox). -*NOTE:* This does not dictate that dune must maintain the equivalent of -default switches, or predetermine any other implementation choice. But the -support for tool management must be designed in way that makes it simple and -reliable to use installed tools outside of a project sandbox (e.g., by -adding the location of a directory of binaries to their `PATH` or some other +*NOTE:* This does not dictate that dune must maintain the equivalent of default +switches, or predetermine any other implementation choice. But the support for +tool management must be designed in way that makes it simple and reliable to use +installed tools outside of a project sandbox (e.g., by adding the location of a +directory of binaries in a workspace to their system `PATH`, or some other means).
@@ -385,17 +385,6 @@ installed, dune subcommands must produce a clear user error explaining to users that the tool is not available and directing them to install it as a dune managed tool. -###### 2.3.1.1.2. When package management is NOT enabled in a workspace - -When package management is enabled in a workspace and a tool is not installed -(e.g., `.ocamlformat`), dune subcommands (such as `dune fmt`, `dune build @doc`, -or `dune utop`) should fall back to the system PATH. - -**Note** This is motivated by integration with editor developers who would like -a single point of truth for running tools, and for dune to handle it. This would -mean opam users can continue to use dune in which ever way they please and the -editors will not have to care. -
Motivation and context @@ -413,6 +402,17 @@ Related issues:
+###### 2.3.1.1.2. When package management is NOT enabled in a workspace + +When package management is enabled in a workspace and a tool is not installed +(e.g., `.ocamlformat`), dune subcommands (such as `dune fmt`, `dune build @doc`, +or `dune utop`) should fall back to the system PATH. + +**Note** This is motivated by integration with editor developers who would like +a single point of truth for running tools, and for dune to handle it. This would +mean opam users can continue to use dune in which ever way they please and the +editors will not have to care. + #### 2.4. Project dependency tools When a tool is installed as a *project dependency* (under any qualification), From cc4cb527fc58b06d25675349be1e8d7dee379e08 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Sat, 18 Jul 2026 01:49:56 -0400 Subject: [PATCH 62/72] Flesh out integration axis and improve organization Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 173 ++++++++++++++++++++------- 1 file changed, 128 insertions(+), 45 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index f71f9736bde..fd8e5eaa3df 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -14,7 +14,7 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni **Table of Contents** -- [Tools Requirements like](#tools-requirements-like) +- [Dune Tools Requirements](#dune-tools-requirements) - [Summary](#summary) - [How to Read This Document](#how-to-read-this-document) - [Terminology](#terminology) @@ -46,10 +46,12 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [2.3.1.1.2. When package management is NOT enabled in a workspace ](#23112-when-package-management-is-not-enabled-in-a-workspace) - [2.4. Project dependency tools](#24-project-dependency-tools) - [3. Dependency and Integration](#3-dependency-and-integration) - - [3.1 Integration spectrum](#31-integration-spectrum) - - [3.1. Compiler compatibility](#31-compiler-compatibility) - - [3.2. Dependency isolation](#32-dependency-isolation) - - [4.2 Dependency spectrum](#42-dependency-spectrum) + - [3.1 Integration axis](#31-integration-axis) + - [3.1.2. Tool isolation (I1)](#312-tool-isolation-i1) + - [3.1.2 Compiler integrations (I2)](#312-compiler-integrations-i2) + - [3.1.2.1 Identifying compiler integrations](#3121-identifying-compiler-integrations) + - [3.1.2.2 Handling compiler integration constraints](#3122-handling-compiler-integration-constraints) + - [4.2 Dependency axis](#42-dependency-axis) - [4.1. Discretionary tools (D1)](#41-discretionary-tools-d1) - [4.1.1. Not build triggers](#411-not-build-triggers) - [4.1.2. Cannot be referenced in build rules TODO](#412-cannot-be-referenced-in-build-rules-todo) @@ -61,7 +63,8 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [4.2.1. Must always be installed TODO](#421-must-always-be-installed-todo) - [5. UI](#5-ui) - [5.1. CLI](#51-cli) - - [5.2. Persistent configuration](#52-persistent-configuration) + - [5.1.1. Managing multiple tools](#511-managing-multiple-tools) + - [5.2. Persistent configuration for discretionary tools](#52-persistent-configuration-for-discretionary-tools) - [5.4. Tool references in actions](#54-tool-references-in-actions) - [5.5. Legacy migration](#55-legacy-migration) - [TODO Orphans to put in proper section](#todo-orphans-to-put-in-proper-section) @@ -152,19 +155,32 @@ The environments within which a tool are available must be scoped. It must be possible to install tools per-workspace, such that workspaces define a sub-environment. I.e., each workspace has its own isolated tool installations -that don't affect other workspaces or interfere the system. +that don't affect other workspaces or interfere with software on the host system. See [Directory structure](./implementation.md#directory-structure) for storage locations. #### 1.2.2. Dune context scope -CR Shon: Is this actually a good idea? How do tools interact with contexts? - It must be possible to install the versions of a tool per-dune context within a workspace, such that different contexts can use different versions of tools. +TODO: tools as lock dirs to impl doc. In particular, specifying multiple tools +in field in one stanza, to avoid lots of duplication, but mapping this to +separate lock dirs. + +
+ +Motivation and context + + This follows the example of `dune pkg`'s `lock_dir` in the context stanza, and -allows developers to set up different tooling configurations (CR. What for?). +allows developers to set up different tooling configurations. + +We have sketched a preliminary design that considers `tools` stanzas as +specialized form of `lock_dir` stanza that inherits the fields of the active +`lock_dir` in a context if no overriding fields are specified. + +
#### 1.2.3. System wide scope @@ -423,62 +439,129 @@ discretionary tools and project dependency tools in the path. ### 3. Dependency and Integration -Tools are used in numerous ways in projects and for ad hoc developer needs, and -they require different levels of integration and interdependence with the other -components (tools, libraries, or other artifacts) of a workspace. We can -position these varieties of difference on a coordinate system with two axes: +Tools are depended on in numerous ways to build and develop projects and they +require different levels of integration and interdependence with the other +components (tools, libraries, or other artifacts) of a workspace. Dune must be +able to manage all installable tools across this variety, otherwise the user +experience of tool management will feel inconsistent and irregular, and users +will inevitably find certain subsets of tools unavailable or requiring +unexpected workarounds creating a fragmented and awkward user experience. + +We can further refine this requirement along two axes. -#### 3.1 Integration spectrum +#### 3.1 Integration axis Tools lie along a spectrum of integration requirements with other dependencies -in the project, which we can indicate with these tree points: +in the project, which we can indicate with tree points: - I1: At the minimum extreme, some tools can be solved and built in complete - isolation from the rest of the project they are used in. E.g., ocamlformat. + isolation from the rest of the project they are used in. Tools of this sort + include ocamlformat, ocp-indent, dune-release, and opam-publish. + - I2: In the midpoint, some tools must integrate with a subset of a project's - dependencies. E.g., ocamllsp must use the same compiler version. -- I3: At the maximum extreme, some tools must be built within the entire - dependency context of the project. E.g., menhir when used together with its - runtime libraries, to ensure that the generated parser code is compatible with - the runtime library. + dependencies. The most common examples in this space are tools that must + integrate with the compiler, such as ocamllsp, odoc, or merlin, but don't need + to integrate with other libraries in the workspace. Some tools also integrate + only with select components other than the compiler, such as menhir and atd. + These tools require integration with the particular version of their runtime + libraries in certain modes of use, but don't require integrations otherwise, + and since the tools themselves have a wired dependency cone than the runtime + libraries, it could sometimes be helpful to bud the executables in a separate + dependency context, pinned only to the needed runtime library version (e.g., + to avoid conflicts over a CLI parser library). + +- I3: At the maximum extreme, some tools could require be built within the + entire dependency context of the project. We are not aware of any tools that + require this currently, but we can consider utop is an illustrative example, + since its own dependencies (such as `lwt`, `xdg`, and `logs`) need to + integrate with the versions in the environment it is installed in, and it + cannot load code into the top level that would require differing on these + versions. + +Tools lying along I1 and I2 must be supported with elegant solutions where +coupling is required, and the most possible orthogonality in features when it is +not. + +##### 3.1.1. Tool isolation (I1) + +Unnecessary coupling between tools and other dependencies of a project should +not be introduced. When an *installable* tool sits at I1 on the integration +axis, dune should be able to install it in a workspace, without regard to any +possible conflicts with *project dependencies*. By definition, tools at I1 have +no integration requirements with other components in a project, so there are no +grounds for such conflict. This property *should* hold rather than *must*, +because it may be infeasible and unnecessary to implement this requirement for +tools that are project dependencies. + +However, for discretionary tools at I1 this property *must* be guaranteed. -Tools lying along the entire spectrum must be supported elegant solutions where -coupling is required, and the most possible orthogonality in features when it is not. +
+ +Motivation and context + -##### 3.1. Compiler compatibility +See [Directory structure](./implementation.md#directory-structure) for lock +directory locations. + +
+ +###### 3.1.1.1 Optimal builds -TODO: need to handle tools that are compiler integrations, and want to have an "optimal" dependency environment. -E.g., installing the latest version of a tool, with whatever compiler version it may need, or installing a tool that is not compatible with your env compiler. +Dune should not needless compile or rebuild dependencies that can be shared +without conflict. E.g., if the needed version of a discretionary tool +can be installed by reusing the compiler version already installed for a +workspace, or by pulling it from the cache, this should be preferred over +rebuilding the tool or its dependencies from scratch. -By default, tools are built with a compiler matching the environment. This -ensures tools like ocamllsp can read project build artifacts correctly. +##### 3.1.2 Compiler integrations (I2) -Users should be able to opt out of compiler matching per-tool for tools that -don't need it (e.g., formatters that only parse source text). +TODO: `integrates_with compiler|minhirLib` ? +TODO: Need to spec more general integration requirements, with compiler as +special case. -CR-soon Alizter: The utility of opting out seems debatable, but is part of the -package-tool continuum approach. +Tools that integrate with the compiler hold a special status among tools, as a +result of their ubiquitous use and the special position of the compiler itself. +The most widely used tools of this sort are ocamllsp and odoc. Dune must provide +robust, intuitive, and flexible support for managing these tools. + + +
+ +Motivation and context + See [Compiler matching](./implementation.md#compiler-matching) for the detection algorithm. -CR-someday Alizter: Reverse influence - could tools constrain the project? E.g., -if ocamllsp isn't available for OCaml 5.3 yet, a user might want their tool -requirements to influence which compiler version they use. Currently tools are -fully isolated and don't affect project solving. This would be a significant -departure from the orthogonality principle. +
-##### 3.2. Dependency isolation +###### 3.1.2.1 Identifying compiler integrations -The dependencies of each tool should be solved independently with its own lock -directory. Tool dependencies do not affect the project's `dune.lock`, and vice -versa. +It must be possible to identify tools that require integration with the compiler +version in a workspace and install them appropriately (e.g., via an explicit +designation provided by the user, a list of known tools, some data available in +the package definition, or any other means). -See [Directory structure](./implementation.md#directory-structure) for lock -directory locations. +###### 3.1.2.2 Handling compiler integration constraints + +When a user has specified the intent to install a compiler integration that is +incompatible with the compiler version installed in the workspace, dune must +handle the conflict gracefully and with clear guidance to the user. + +This could mean automatically downgrading the compiler (if permitted by +the constraints), but a user error with clear instructions is probably just as +effectively, less surprising, and much easier to implement. E.g., "tool t cannot +be installed at version v because it requires compiler <= n, but compiler m is +currently installed. Add the following qualified constraint to your project +dependencies: `(ocaml (and (>= 5.1 (or (not :with-dev-setup) (< 5.5))))`" etc. + +Regardless of the implementation approach taken, it must be possible to +constrain the compiler versions to work with the desired integrations during +development without those constraints polluting the constraints of packages in +the workspace when they are installed. -#### 4.2 Dependency spectrum +#### 4.2 Dependency axis Tools lie along a spectrum of dependency status (i.e., to what extent they are dependencies for the project) which we can indicate with these tree points: From 456df00681edefdd779f9edac0820139467d79b8 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Sat, 18 Jul 2026 17:52:35 -0400 Subject: [PATCH 63/72] Fill in motivation for dependency isolation Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 38 +++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index fd8e5eaa3df..7694cb01f01 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -484,7 +484,7 @@ not. ##### 3.1.1. Tool isolation (I1) -Unnecessary coupling between tools and other dependencies of a project should +Unnecessary coupling between tools and other dependencies of a project _should_ not be introduced. When an *installable* tool sits at I1 on the integration axis, dune should be able to install it in a workspace, without regard to any possible conflicts with *project dependencies*. By definition, tools at I1 have @@ -493,15 +493,45 @@ grounds for such conflict. This property *should* hold rather than *must*, because it may be infeasible and unnecessary to implement this requirement for tools that are project dependencies. -However, for discretionary tools at I1 this property *must* be guaranteed. +However, for *discretionary tools* at I1 this property *must* be guaranteed: +dune must support installation of any installable, discretionary tool at I1, +without regard to possible dependency conflicts with project dependencies.
Motivation and context -See [Directory structure](./implementation.md#directory-structure) for lock -directory locations. +To indicate just one dimension of the motivations here, consider this small set +of arbitrarily chosen packages providing I1 tools, which are all currently +published to opam and all currently incompatible with ocaml 5.5 (due to direct +or transitive constraints): + +- [cca](https://ocaml.org/p/cca/latest): "A framework for differential source + code analyses" +- [comby](https://ocaml.org/p/comby/latest): "A tool for structural code search + and replace that supports ~every language" +- [crs](https://ocaml.org/p/crs/latest): "A tool for managing inline review + comments embedded in source code" +- [depgraph](https://ocaml.org/p/depgraph/latest): "dot graphs out of ocamldep + output" +- [electrod](https://ocaml.org/p/electrod/latest): "Formal analysis for the + Electrod formal pivot language" +- [facteur](https://ocaml.org/p/facteur/latest): "Tool to send an email" +- [pfff](https://ocaml.org/p/pfff/latest): "Tools and APIs for program analysis, + code visualization, refactoring" +- [pgn_to_tex](https://ocaml.org/p/pgn_to_tex/latest): "A chess PGN to TeX conversion tool" +- [rdr](https://ocaml.org/p/rdr/latest): "Rdr is a cross-platform binary + analysis and reverse engineering tool, utilizing a unique symbol map for + global analysis." + +Without support for dependency isolation, users would be forced to chose between +avoiding recent compiler versions or making use of these (and many other) +available tools. This would violate the design principles of orthogonality and +generality, and yield and necessarily limited usability. + +See [Directory structure](./implementation.md#directory-structure) for proposed +lock directory locations.
From a39ccc37923b9eca83eb175f132f9d60b9998e1a Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Sat, 18 Jul 2026 17:55:24 -0400 Subject: [PATCH 64/72] Fix wording for optimal bluilds req Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 7694cb01f01..1baa9b6bc13 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -538,10 +538,10 @@ lock directory locations. ###### 3.1.1.1 Optimal builds Dune should not needless compile or rebuild dependencies that can be shared -without conflict. E.g., if the needed version of a discretionary tool -can be installed by reusing the compiler version already installed for a -workspace, or by pulling it from the cache, this should be preferred over -rebuilding the tool or its dependencies from scratch. +without conflict. E.g., if the needed version of an I1 discretionary tool can be +installed by reusing the compiler version already used in the workspace, or by +pulling it from the shared cache, this should be preferred over rebuilding the +tool or its dependencies from scratch. ##### 3.1.2 Compiler integrations (I2) From 59452c3fa5d6bc264184214fc65670f1048c800c Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Sat, 18 Jul 2026 18:16:59 -0400 Subject: [PATCH 65/72] Rephrase tool integration requirement Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 48 ++++++++++++++++++---------- 1 file changed, 31 insertions(+), 17 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index 1baa9b6bc13..eb032529f2c 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -543,36 +543,50 @@ installed by reusing the compiler version already used in the workspace, or by pulling it from the shared cache, this should be preferred over rebuilding the tool or its dependencies from scratch. -##### 3.1.2 Compiler integrations (I2) +##### 3.1.2 Tool integration (I2) -TODO: `integrates_with compiler|minhirLib` ? -TODO: Need to spec more general integration requirements, with compiler as -special case. - -Tools that integrate with the compiler hold a special status among tools, as a -result of their ubiquitous use and the special position of the compiler itself. -The most widely used tools of this sort are ocamllsp and odoc. Dune must provide -robust, intuitive, and flexible support for managing these tools. +When a tool requires integration with other components in the workspace at their +specific version, it must be possible to specify this requirement and have the +solution, build, and install enable that needed integration. +Tools that integrate with the compiler are a very special case of I2 tools +requiring this need, and they have an important status among other tools because +some are used ubiquitous and because he compiler itself has a special position +in the dependency tree of any OCaml project. The most widely used tools of this +sort are ocamllsp and odoc. Dune must provide robust, intuitive, and flexible +support for managing these tools.
Motivation and context +One way to address this may be thru the an equivalent of the `constraints` field +in the `lock_dir` stanza, where an absent dep-specification indicated the need +to use the same version in the context's active `lock_dir`. E.g., as + +``` lisp +(tools + ((ocamllsp (from ocaml-lsp-server)) + utop) + (constraints ocaml)) +``` + +Which would require that the named tools be constrained on the version of ocaml +used in the workspace. + +A special purpose field could also be introduced for this purpose. + +We can also consider data added to opam files that allows this requirement to be +specified for provided tools at the package data level, instead of forcing this +upon consuming users. + See [Compiler matching](./implementation.md#compiler-matching) for the detection algorithm.
-###### 3.1.2.1 Identifying compiler integrations - -It must be possible to identify tools that require integration with the compiler -version in a workspace and install them appropriately (e.g., via an explicit -designation provided by the user, a list of known tools, some data available in -the package definition, or any other means). - -###### 3.1.2.2 Handling compiler integration constraints +###### 3.1.2.1. Respecting integration constraints When a user has specified the intent to install a compiler integration that is incompatible with the compiler version installed in the workspace, dune must From de8f2fa26d9c55b15e40c49fe765dbe258fdc432 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Sat, 18 Jul 2026 18:21:18 -0400 Subject: [PATCH 66/72] Generalize respecting integration constraints Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index eb032529f2c..b59601bb2da 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -535,7 +535,7 @@ lock directory locations.
-###### 3.1.1.1 Optimal builds +###### 3.1.1.1. Optimal builds Dune should not needless compile or rebuild dependencies that can be shared without conflict. E.g., if the needed version of an I1 discretionary tool can be @@ -543,7 +543,7 @@ installed by reusing the compiler version already used in the workspace, or by pulling it from the shared cache, this should be preferred over rebuilding the tool or its dependencies from scratch. -##### 3.1.2 Tool integration (I2) +##### 3.1.2. Tool integration (I2) When a tool requires integration with other components in the workspace at their specific version, it must be possible to specify this requirement and have the @@ -565,7 +565,7 @@ One way to address this may be thru the an equivalent of the `constraints` field in the `lock_dir` stanza, where an absent dep-specification indicated the need to use the same version in the context's active `lock_dir`. E.g., as -``` lisp +``` (tools ((ocamllsp (from ocaml-lsp-server)) utop) @@ -588,9 +588,13 @@ algorithm. ###### 3.1.2.1. Respecting integration constraints -When a user has specified the intent to install a compiler integration that is -incompatible with the compiler version installed in the workspace, dune must -handle the conflict gracefully and with clear guidance to the user. +When a user has specified the intent to install an I2 tool that is incompatible +with the required packages already installed in the workspace, dune must handle +the conflict gracefully an with clear guidance to the user. + +E.g., if a user tries to install the compiler integration `ocamllsp` and not +version is compatible with the compiler version already installed in the +workspace, dune must report and/or solve the conflict cleanly. This could mean automatically downgrading the compiler (if permitted by the constraints), but a user error with clear instructions is probably just as From 7c366fc8c8f4b99f4ec7a3169c3550f62efcd987 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Sat, 18 Jul 2026 23:44:27 -0400 Subject: [PATCH 67/72] Finish cleaning up organization and fleshing out spectra Signed-off-by: Shon Feder --- doc/dev/tools-design/requirements.md | 248 ++++++++++++--------------- 1 file changed, 106 insertions(+), 142 deletions(-) diff --git a/doc/dev/tools-design/requirements.md b/doc/dev/tools-design/requirements.md index b59601bb2da..df18839b00e 100644 --- a/doc/dev/tools-design/requirements.md +++ b/doc/dev/tools-design/requirements.md @@ -47,10 +47,10 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [2.4. Project dependency tools](#24-project-dependency-tools) - [3. Dependency and Integration](#3-dependency-and-integration) - [3.1 Integration axis](#31-integration-axis) - - [3.1.2. Tool isolation (I1)](#312-tool-isolation-i1) - - [3.1.2 Compiler integrations (I2)](#312-compiler-integrations-i2) - - [3.1.2.1 Identifying compiler integrations](#3121-identifying-compiler-integrations) - - [3.1.2.2 Handling compiler integration constraints](#3122-handling-compiler-integration-constraints) + - [3.1.1. Tool isolation (I1)](#311-tool-isolation-i1) + - [3.1.1.1. Optimal builds](#3111-optimal-builds) + - [3.1.2. Tool integration (I2)](#312-tool-integration-i2) + - [3.1.2.1. Respecting integration constraints](#3121-respecting-integration-constraints) - [4.2 Dependency axis](#42-dependency-axis) - [4.1. Discretionary tools (D1)](#41-discretionary-tools-d1) - [4.1.1. Not build triggers](#411-not-build-triggers) @@ -437,6 +437,16 @@ provided for executing discretionary tools. E.g., a mechanism like `dune env` that would make the path to dune-managed binaries available must include both discretionary tools and project dependency tools in the path. +#### 2.4. Dog fooding + +Tools must work in the dune repository itself. Dune developers should be able to +run `dune tools add ocamlformat` and `dune tools add ocaml-lsp-server` when +working on dune. + +This is enabled by the orthogonality design (3.2): tools are solved and built +independently from project dependencies, so they don't require a working project +lock directory. + ### 3. Dependency and Integration Tools are depended on in numerous ways to build and develop projects and they @@ -628,25 +638,95 @@ dependencies for the project) which we can indicate with these tree points: generation, they could be qualified project dependencies behind a hypothetical `with-gen` filter). +Note that, since tools like `utop`, `odoc`, and `ocamlformat` have dedicated +builtin rules from dune, they are technically not D1. + ##### 4.1. Discretionary tools (D1) +Dune must be able to manage discretionary tools. + ###### 4.1.1. Not build triggers -Locking or adding tools in at D1 level tools must not trigger project builds. +Locking or adding discretionary tools must not trigger project builds. + +###### 4.1.2. Cannot be referenced in build rules + +Discretionary tools managed by dune must not be referenced in user build rules: dune +must not add these binaries to its path when running rules, and must not +resolve `%{bin:...}` forms that attempt to reference them. + +By definition, tools referenced in rules are at least D2. + +###### 4.1.2.1. Useful guidance on invalid reference in rules + +When a discretionary tool is configured and an invalid reference to it is found +in a build rule, dune should report an error with clear guidance to users, +advising them to move the tool configuration into the appropriate package +dependency. + +###### 4.1.3. Subset of D2 and D3 tools functionality + +The functionality of discretionary, D1 tools must a strict subset of the functionality over +D2 and D3 project dependency tools: anything you can do with + +##### 4.2. Qualified project dependency tools (D2) + +Qualified project dependency tools must be installable based on the existing +package dependency configuration mechanism. This configuration may be extended +to support tool-specific configuration when the need and means to achieve that +is known. + +
+ +Motivation and context + + +When tools are invoked in build rules, they are dependencies of a project, since +the full breadth of the project cannot be built without installing the tool. We +have an existing mechanism for declaring dependencies for installation, and so +based on our principles of orthogonality and complementarity, we must not +introduce redundant ways of solving the same problem that is already accounted +for. Doing so would not only be inelegant, but lead to user confusion and +package specification fragmentation, since the existing systems is also +supported by opam. + +
+ +###### 4.2.1. Usable as tools -###### 4.1.2. Cannot be referenced in build rules TODO +Qualified project dependency tools must be usable in exactly the same ways as +discretionary, D1 tools. E.g., if D1 tools can be run via a command like `dune +tools exec ...` then so too must be D2 tools. However, the implementation is +not prescribed by the requirement, and we could instead permit D1 tools to be +run via the existing `dune exec ...` subcommand, providing a simpler and more +intuitive interface to users than requiring to separate subcommands. -##### 4.2. Qualified dependency tools (D2) TODO +###### 4.2.2. Installable via qualification -###### 4.2.1. Must be usable as tools +If a set of tools are qualified with a filter such as `:with-test` or +`:with-doc`, it must be possible to install just that set together (in addition +to the unqualified dependencies). -###### 4.2.1. Must be installable via qualification +###### 4.2.3. Builtin D2 tools + +A select subset of keystone tools are treated by dune as builtin qualified +dependencies, including `odoc`, `utop`, and `ocamlformat` as the most widely +used. These should be treated as if they have qualified dependencies built in, +with further constraint or specification of how to install them available as a +user override on top of the default configuration. ##### 4.3. Unqualified dependency tools (D3) TODO -###### 4.3.1. Must be usable as tools TODO +###### 4.3.1. Usable as tools + +As with 4.2.1, unqualified project dependency tools must be usable in exactly +the same ways as D1 and D2 tools. + +###### 4.2.1. Installed on build -###### 4.2.1. Must always be installed TODO +Unqualified tools must be installed when a project is built. + +(Already satisfied by the current implementation of package managemet.) #### 5. UI @@ -668,8 +748,8 @@ operations](./implementation.md#batch-operations) for batch commands. ##### 5.1.1. Managing multiple tools -It must be possible to change (e.g., update, install, remove) the installation -of all configured tools by issuing a single command. +It must be possible to change (e.g., update, install, or remove) the +installation of a set of configured tools by issuing a single command.
@@ -690,6 +770,19 @@ Related issues:
+##### 5.1.2. Watch mode integration + +Tool operations (e.g., `dune tools add`, `run`, etc.) must work correctly when a +watch server is running (`dune build -w`). Rather than directly manipulating +lock directories, tool commands should coordinate with the watch server via RPC +to avoid races and ensure the server picks up newly added tools. + +##### 5.1.3 Avoid invocation collisions + +When a `dune tools` command is run that depends on the state of the workspace, +it should not interfere with concurrent running dune commands or lead to invalid +results or states. This may just mean refusing to run if the build directory is +locked, cooperating to sequence requests, or something else. #### 5.2. Persistent configuration for discretionary tools @@ -711,132 +804,3 @@ See [The `(tool)` stanza](./implementation.md#the-tool-stanza) for proposed synt
-#### 5.4. Tool references in actions - -CR Shon: anything referenced by an action is U2 or greater on the "integration -requirement spectrum". - -Build actions (both user-written rules and dune's internal rules like -formatting) must be able to reference the installed tools. E.g., via -`%{bin:...}` pforms, the base executable name, (or by some new mechanism, if -there is need for it). - -#### 5.5. Legacy migration - -TODO: is this just a task for our project, or is it actually a requirement? Is the requirement about backward compat. - -The legacy `.dev-tools.locks/` system must be removed and replaced by this -design. Entailing updating all documentation and communicating needed users -interventions, as necessary. - - - - - - - - - -# TODO Orphans to put in proper section - - -#### N. Watch mode integration - -Tool operations (e.g., `dune tools add`, `run`, etc.) must work correctly when a -watch server is running (`dune build -w`). Rather than directly manipulating -lock directories, tool commands should coordinate with the watch server via RPC -to avoid races and ensure the server picks up newly added tools. - -CR-soon Alizter: Specify the RPC protocol for tool operations. What messages are -needed? How does the watch server respond to tool additions? - -CR-soon Alizter: Document concurrent access behavior. What happens if two -terminals run `dune tools add` simultaneously? Or `add` while `run` is building? - -#### 4. Dog fooding - -Tools must work in the dune repository itself. Dune developers should be able to -run `dune tools add ocamlformat` and `dune tools add ocaml-lsp-server` when -working on dune. - -This is enabled by the orthogonality design (3.2): tools are solved and built -independently from project dependencies, so they don't require a working project -lock directory. - - -# TODO (Shon, Ambre, Sudha): Analyze out the axes of use and integration - -This is about "the tool-dependency spectrum": most tools do not fit into the -binary between: - -- an adhoc binary a person wants to run a few times or -- a library dependency required for a project - -Most tools actually have fall on a spectrum with varying levels of integration -requirements, and various kinds of usage within the SDLC. - -Three ways of invoking a tool: - -- shell environment: (dune tools env) -- sub commands: dune fmt / dune build @doc -- dedicated dune runner subcommands: dune tools run/exec - -How do tools that are required for development currently interact with the - current deps? E.g., from opam, `(deps (odoc (and 2.1.1 :with-dev-setup)))` - -## There are at least 2 axes - -Each a spectrum, but we can sketch its shape using the two extremes and the -mixed case. - -### Usage categories spectrum: - -- U1. tools we want to install and run on an adhoc basis: dune tools install og; - og foo `dune build @tools` `dune build @editors-tools` :tools `dune pkg lock`. -- U2. tools we want to use as part of a build alias, but not as part of the - package dependencies (for tests, for build docs, fmt, could be geneneralized to - any build alias): with-doc, with-test, with-dev-setup -- U3. tools that are required for a build (e.g., menhir or atd), and just need - to be a package dependency unconditionally (but which users will also want to - be able to execute)! - -### Integration requirements spectrum: - -- I1. tools that must integrate with all other packages in a solution (e.g., - because of shared libraries) -- I2. tools that must integrate just with the compiler (aka "compiler - integrations" or "toolchain utilities") -- I3. tools that do not need to integrate with anything, and can be installed in - completely isolated environments. - -## Some example - -pkg foo - deps A.1 - -cmdliner --with-test - deps A.1 - -ocamllsp # exectable, but it hast to have library like integration (with compiler) - -ocamlformat - deps A.2 - -atd -> foo_j.ml (* as a fixuter json serde *) - deps A.3 - -## Some ideas - -Could consider a qualifier on executable packages - -- like "disjoint" or "sandboxed": but would have to be only for executables -- :exec-only, in which case (so long as not compiler integration!) can always be sandboxed - -Libs vs. executables. - -TODO: Consider use cases in many different forms and classify. - -- Menhir -- Ocsigen -- Coq/Fstar -- grep tools From 380163500627ab09b9dae4f479b5e919bc1c7015 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Sat, 18 Jul 2026 23:45:40 -0400 Subject: [PATCH 68/72] Remove implementation spec To be added back in a new branch, and staged as its own PR. Signed-off-by: Shon Feder --- doc/dev/tools-design/implementation.md | 1581 ------------------------ 1 file changed, 1581 deletions(-) delete mode 100644 doc/dev/tools-design/implementation.md diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md deleted file mode 100644 index 9258a90fbda..00000000000 --- a/doc/dev/tools-design/implementation.md +++ /dev/null @@ -1,1581 +0,0 @@ -## Tools Implementation - -Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) - -## Summary - -This document specifies the implementatino of dune's tool management system. - - - - -**Table of Contents** - -- [Tools Implementation](#tools-implementation) -- [Summary](#summary) -- [Design](#design) - - [The `(tool)` stanza](#the-tool-stanza) - - [Syntax](#syntax) - - [Fields](#fields) - - [Compiler matching](#compiler-matching) - - [Examples](#examples) - - [Invalid tool selections](#invalid-tool-selections) - - [CLI commands](#cli-commands) - - [Version syntax](#version-syntax) - - [Batch operations](#batch-operations) - - [Open question: version separator](#open-question-version-separator) - - [Open question: `add` vs `install` naming](#open-question-add-vs-install-naming) - - [Open question: `dune tools run` PATH fallback](#open-question-dune-tools-run-path-fallback) - - [Tool resolution](#tool-resolution) - - [Source resolution](#source-resolution) - - [Version selection](#version-selection) - - [Example scenarios](#example-scenarios) - - [Directory structure](#directory-structure) - - [Tool pforms TODO](#tool-pforms-todo) - - [Syntax](#syntax-1) - - [Resolution](#resolution) - - [Examples](#examples-1) - - [Future: library access](#future-library-access) -- [Relationship to Package Management](#relationship-to-package-management) - - [Orthogonality: a key design principle](#orthogonality-a-key-design-principle) - - [The package-tool continuum](#the-package-tool-continuum) - - [Key differences from project dependencies](#key-differences-from-project-dependencies) - - [Why tools don't use `dune.lock/`](#why-tools-dont-use-dunelock) - - [Locking vs building](#locking-vs-building) - - [Future: Relocatable compiler and caching](#future-relocatable-compiler-and-caching) - - [Potential future stanza options](#potential-future-stanza-options) -- [GitHub Issues Addressed](#github-issues-addressed) - - [Arbitrary tool packages ([#12913])](#arbitrary-tool-packages-12913) - - [Lock directory clutter ([#10955], [#12097])](#lock-directory-clutter-10955-12097) - - [Compiler ABI mismatch ([#11229])](#compiler-abi-mismatch-11229) - - [Graceful fallback for missing tools ([#10578])](#graceful-fallback-for-missing-tools-10578) - - [Version constraints ([#12777])](#version-constraints-12777) - - [Version consistency ([#10688])](#version-consistency-10688) - - [Dev tools redesign ([#12914])](#dev-tools-redesign-12914) - - [Tool dependency isolation ([#12551])](#tool-dependency-isolation-12551) - - [No project build triggers ([#11037])](#no-project-build-triggers-11037) - - [Compiler-independent formatters ([#11038])](#compiler-independent-formatters-11038) - - [Version from config files ([#5315])](#version-from-config-files-5315) - - [Version constraint parsing ([#12866])](#version-constraint-parsing-12866) - - [Opam with-dev-setup integration ([#12135])](#opam-with-dev-setup-integration-12135) - - [Helpful error messages ([#13235], [#12975])](#helpful-error-messages-13235-12975) - - [Batch installation ([#12557])](#batch-installation-12557) - - [Failure isolation ([#12818])](#failure-isolation-12818) - - [Utop findlib integration ([#13471])](#utop-findlib-integration-13471) -- [Comparison with Other Tools](#comparison-with-other-tools) - - [uv (Python)](#uv-python) - - [cargo (Rust)](#cargo-rust) - - [cargo-run-bin (Rust community tool)](#cargo-run-bin-rust-community-tool) - - [npm (JavaScript)](#npm-javascript) - - [How Our Design Compares](#how-our-design-compares) - - [Tools compared](#tools-compared) - - [Comparison table](#comparison-table) - - [Project-local](#project-local) - - [Multi-version](#multi-version) - - [Declarative config](#declarative-config) - - [Compiler matching](#compiler-matching-1) - - [PATH fallback](#path-fallback) - - [Precompiled binaries](#precompiled-binaries) - - [Tool isolation](#tool-isolation) - - [Ephemeral runs](#ephemeral-runs) - - [Batch install](#batch-install) - - [Binary selection](#binary-selection) - - [Tool discovery](#tool-discovery) - - [Tool upgrade](#tool-upgrade) - - [System-wide install](#system-wide-install) - - [Recommendations from Research](#recommendations-from-research) -- [Learnings from the Prototype](#learnings-from-the-prototype) - - [What Worked Well](#what-worked-well) - - [Problems Discovered](#problems-discovered) - - [Reference Packages for Testing](#reference-packages-for-testing) -- [Ideas](#ideas) - - [Editor integration](#editor-integration) - - - -## How to Read This Document - -[Design](#design) details _how_ the system implements the requirements: -stanza syntax, CLI commands, version resolution algorithm, and directory -structure. Each section notes which requirements it implements. Open questions -are marked where decisions are pending. - -[Relationship to Package Management](#relationship-to-package-management) -explains the orthogonality principle and how tools differ from project -dependencies. - -[Comparison with Other Tools](#comparison-with-other-tools) analyzes how -`uv`,`cargo`, `cargo-run-bin`, and `npm` handle tool management, informing our -design decisions. - -## Design - -### The `(tool)` stanza - -_Implements: [1.3 Version specification](./requirements.md#14-version-specification), -[1.6 Binary selection](./requirements.md#16-binary-selection), -[3.1 Compiler compatibility](./requirements.md#31-compiler-compatibility), -[4.2 Persistent configuration](./requirements.md#42-persistent-configuration)_ - -Tools are declared in `dune-workspace` using the `(tool)` stanza. The stanza -uses the same dependency syntax as `(depends ...)` for the package field. - -#### Syntax - -```lisp -(tool - (package ) ; Required: package name with optional constraint - (executable ) ; Optional: binary name, defaults to package name - (skip_compiler_match) ; Optional: flag to disable compiler matching - (repositories ...)) ; Optional: restrict to specific repositories -``` - -CR-soon Alizter: missing pin - -CR-soon Alizter: Should share some of lock_dir stanza fields. Need to do a -comparison and explain why it is or is not a good idea. - -CR-soon Alizter: Edge cases for stanza parsing: - -- Duplicate stanzas for same package with different constraints: error, or - merge? -- Invalid package name syntax: error message? -- Stanza references non-existent repository: when to error (parse time vs solve - time)? - -CR-soon Sudha247: Capture constraints here? Related issue: -[dune#12777](https://github.com/ocaml/dune/issues/12777) - -#### Fields - -- **`(package )`** (required): The opam package providing the tool. Accepts - either a plain package name or a name with version constraint: - - Plain: `ocamlformat` - - With constraint: `(ocamlformat (= 0.26.2))` - - Version operators: `=`, `<>`, `<`, `>`, `<=`, `>=` - - Conjunctions: `(and (>= 0.25.0) (< 0.27.0))` - -- **`(executable )`** (optional): The binary to run. Defaults to the - package name. Required when the package provides multiple binaries. - -- **`(skip_compiler_match)`** (optional): When present, disables the default - compiler matching behavior. Use this for tools that don't need compiler - compatibility (e.g., formatters that only parse source text). - -CR-soon Alizter: The prototype uses `compiler_compatible` (opt-in) but this doc -says `skip_compiler_match` (opt-out). Reconcile naming and semantics. - -- **`(repositories )`** (optional): Restrict package resolution to the - named repositories. Useful for binary package repositories. - -CR-soon Alizter: The `(repositories)` field is parsed but verify that the -restriction is actually applied during solving in the prototype. - -CR-soon Alizter: Missing pins - -#### Compiler matching - -_Implements: [3.1 Compiler compatibility](./requirements.md#31-compiler-compatibility)_ - -Tools that read project build artifacts (e.g., ocamllsp reading `.cmi` files) -must be built with a compatible compiler to avoid ABI mismatches. By default, -tools are built with a compiler matching the environment. - -The compiler is detected in this order: - -1. **Project compiler** - if using dune package management with a `dune.lock/` -2. **System OCaml** - from opam switch or PATH -3. **Any compatible version** - if no compiler is detected - -When `(skip_compiler_match)` is set, the tool is solved without compiler -constraints, allowing it to use any OCaml version. - -CR-soon Alizter: Edge cases for compiler matching: - -- No compiler found (no opam switch, no system OCaml): error, or pick any? -- System OCaml version changes between `add` and `run`: rebuild, or warn? -- Tool requires newer OCaml than project: error message? - -#### Examples - -```lisp -;; Minimal: just the package name -(tool - (package ocamlformat)) - -;; With version constraint -(tool - (package - (ocamlformat - (= 0.26.2)))) - -;; With version range -(tool - (package - (ocamlformat - (and - (>= 0.25.0) - (< 0.27.0))))) - -;; Specifying the binary for multi-binary packages -(tool - (package menhir) - (executable menhirSdk)) - -;; Tool that doesn't need compiler matching -(tool - (package ocamlformat) - (skip_compiler_match)) - -;; Using a binary package repository -(tool - (package ocaml-lsp-server) - (repositories binary-packages)) -``` - -#### Invalid tool selections - -Some packages cannot be used as tools: - -- **Library-only packages**: Packages that provide no executables (e.g., `base`, - `lwt`) cannot be tools. Error at `dune tools add` time. - -CR-someday Alizter: The `%{tool:lib:...}` pform in [Tool pforms](#tool-pforms) -could enable library-only tool packages, but this needs further design work. - -CR-soon Alizter: How can we error at dune tools add time? We would only find out -after building the tool. - -- **Compiler packages**: Selecting `ocaml-base-compiler` or `ocaml-system` as a - tool is invalid. These conflict with the compiler matching logic. - -- **Dune itself**: Using `dune` as a tool is the "in and out" problem - ([non-requirement 6.3](#63-tools-for-tools)). Circular dependency issues make - this out of scope. - -CR-soon Alizter: The explanations for compiler packages and dune itself need -more thought. - -CR-soon Alizter: Specify exact error messages for these cases. - -### CLI commands - -_Implements: [1.4 Version specification](./requirements.md##14-version-specification), -[1.6 Binary selection](./requirements.md#16-binary-selection), -[2.3 Programmatic use](./requirements.md#23-programmatic-use), [4.1 CLI](./requirements.md#41-cli)_ - -CR-soon Alizter: The CLI commands are generally underspecified. Each command -needs detailed behavior documentation: what triggers downloads/builds, how -errors are reported, interaction with stanzas vs CLI-added tools, exit codes, -output format, etc. - -``` -dune tools List all locked tools (alias for list) -dune tools list List all locked tools and versions - -dune tools add Lock all tools from (tool) stanzas -dune tools add [.] ... Lock specific tool(s) - -dune tools run [.] [--bin ] [-- ] - Run a tool, passing arguments after -- - -dune tools path Print path to the managed binary directory (or directories) -dune tools path [.] [--bin ] Print path to tool executable - -dune tools remove [.] ... Remove a tool's lock directory -``` - -The `--bin` flag is required when a package provides multiple binaries and no -`(executable ...)` is specified in the `(tool)` stanza. - -CR-soon Alizter: Edge cases for `dune tools add`: - -- Package doesn't exist in repository: error message? -- Network unavailable: error message, offline mode? -- Solver fails (unsatisfiable constraints): error with diagnostics? -- Build fails: partial state cleanup? -- Interrupted mid-install: cleanup, or resume on next run? -- Already locked at same version: no-op, or re-solve? -- Already locked at different version: add second version, or replace? - -CR-soon Alizter: The prototype has `--allow-not-installed` flag for -`dune tools path`. Document this flag and its use case (editor integration). - -CR-soon Alizter: `dune tools path` behavior is underspecified: - -- Should `path` trigger download and build if not yet built? Or only return path - if already built, erroring otherwise? -- If `--bin` is specified, can we compute the path without building (since path - is deterministic from package/version/binary name)? -- If package has multiple binaries and no `--bin`: error immediately, or - download and build first to discover available binaries? -- Tool not locked: error, or fall back to `which`? -- How does `path` interact with `(tool)` stanzas vs CLI-added tools? Same - resolution as `run`? - -CR-someday Alizter: Consider a single bin directory with symlinks to all tool -executables (like npm's `node_modules/.bin/`). Instead of per-tool paths, have -`_build/.tools/bin/` containing symlinks to all installed tool binaries. This -simplifies editor integration (one directory to add to PATH), avoids PATH length -limits (especially on Windows), and provides a single stable location for -discovery. Trade-off: need to maintain symlinks as tools are added/removed. -#### Version syntax - -The CLI uses dot-separated version syntax: `.` - -```bash -dune tools add ocamlformat # Latest version -dune tools add ocamlformat.0.26.2 # Specific version -``` - -#### Batch operations - -Batch operations apply to commands that manage tool state, not to commands that -use tools: - -| Command | Batch | Rationale | -| ------------------- | ----- | ----------------------------------------------- | -| `dune tools add` | ✓ | No args: all from stanzas; with args: specified | -| `dune tools remove` | ✓ | Multiple packages | -| `dune tools run` | ✗ | Running is inherently single-target | -| `dune tools path` | ✗ | Path lookup is per-tool | - -CR-soon Alizter: Edge cases for `dune tools remove`: - -- Tool not locked: error, or no-op with warning? -- Tool declared in stanza: remove lock but warn it will be re-added? -- Multiple versions locked: remove all, or require version specifier? -- Remove while tool is running: behavior? - -**`dune tools add`** has two modes: - -```bash -dune tools add # Lock all from (tool) stanzas -dune tools add ocamlformat odoc # Lock specific packages -dune tools add ocamlformat.0.26.2 odoc.2.4.0 # With version constraints -``` - -The no-argument form is critical for CI and onboarding - a single command to set -up all project tooling from declarative configuration. - -CR-someday Alizter: Integration with opam's `:with-dev-setup` marker ([#12135]): - -[#12135]: https://github.com/ocaml/dune/issues/12135 - -Projects often declare their development tools in opam files using the -`:with-dev-setup` filter (e.g., `ocamlformat :with-dev-setup`). Currently, dune -tools doesn't read these - users must manually add each tool or duplicate the -declarations in `(tool)` stanzas. - -One possible approach: - -```bash -dune tools add --with-dev-setup # Install tools from opam :with-dev-setup deps -dune tools add --with-doc # Install tools from :with-doc deps -``` - -This would enable: - -- Single source of truth for project tooling (opam file) -- Automatic tool discovery for new contributors -- CI scripts that don't need to know which tools to install - -CR-soon Alizter: Is `--with-dev-setup` the right approach? Alternatives: - -- Automatic: `dune tools add` (no args) always includes `:with-dev-setup` deps -- Separate command: `dune tools setup` specifically for opam marker integration -- No integration: keep `(tool)` stanzas as the only source, avoid duplication by - not using `:with-dev-setup` in opam files at all The flag approach adds - complexity. Need to evaluate if the use case justifies it. - -CR-soon Alizter: If we do integrate, design questions: - -- Read from `dune-project` `(depends)` or from `.opam` files? -- What if opam file says `ocamlformat` but `.ocamlformat` says `version=0.26.2`? -- How does this interact with `(tool)` stanzas that specify the same package? - -CR-soon Alizter: The prototype requires at least one package argument. The -no-argument batch mode (lock all from stanzas) is not yet implemented. - -#### Open question: version separator - -The current prototype uses `.` as the version separator (`pkg.version`), aligned -with opam conventions. However, `@` is common in other ecosystems: - -- **npm/npx**: `npx cowsay@1.5.0` -- **cargo**: `cargo install ripgrep@14.0.0` - -Opam package names cannot contain dots (allowed: `[a-zA-Z0-9_+-]+`), so `.` is -unambiguous. The choice is primarily about familiarity vs ecosystem consistency. - -#### Open question: `add` vs `install` naming - -The current prototype uses `dune tools add` for adding individual tools. -However, the naming convention is still open for discussion: - -- **Option A**: `add` for individual, `install` for batch (current prototype) -- **Option B**: `install` for both (consistent with legacy, npm, cargo) -- **Option C**: `lock` to emphasize the solving/locking aspect -- **Option D**: Semantic continuum where commands differ by how much work they - do: - - `add` = lock only (solve + write lock) - - `install` = lock + build (download + compile) - - `run` = lock + build + execute This gives users control over when work - happens (useful for CI, Docker builds). - -Considerations: - -- Legacy issues reference `dune tools install` -- `add` suggests incremental, `install` suggests setup -- `lock` aligns with `dune pkg lock` but may confuse (tools also build) -- Option D aligns with user expectations: "add to config" vs "install software" - -#### Open question: `dune tools run` PATH fallback - -Should `dune tools run ` fall back to system PATH when the tool is not -locked? - -- **Option A**: Fall back to PATH (consistent with build rules like `dune fmt`) -- **Option B**: Error with suggestion to lock or install on PATH (current impl) - -Considerations: - -- Falling back makes `dune tools run` a universal "run this tool" command -- Erroring is more explicit and avoids silently using unexpected versions -- Build rules need PATH fallback for gradual adoption (e.g., `dune fmt` without - locking ocamlformat) -- CLI commands could be stricter since the user explicitly requested a tool - -### Tool resolution - -_Implements: [1.5 Multi-version support](./requirements.md#15-multi-version-support), -[2.5 System PATH fallback](./requirements.md#25-system-path-fallback), -[5.1 Format rules](#51-format-rules)_ - -Tool resolution answers two questions: - -1. **Source**: Where does the tool come from? -2. **Version**: Which version of the tool? - -#### Source resolution - -When dune needs a tool, it checks these sources in order: - -1. **Workspace `(tool)` stanza** - declared in `dune-workspace` -2. **Locked versions** - previously added via `dune tools add` -3. **System PATH** - fallback if nothing else found - -The first source that has the tool wins. If no source has it, dune falls back to -system PATH (which may result in "command not found"). - -#### Version selection - -CR-soon Alizter: Version conflict resolution algorithm is undefined. What -happens when multiple sources specify conflicting versions? E.g., stanza says -`(= 0.26.2)` but `.ocamlformat` says `version=0.27.0`. Need to define precedence -or error behavior. - -How the version is determined depends on context: - -**CLI commands** (`dune tools run`, `dune tools add`): - -- Explicit version: `dune tools run ocamlformat.0.26.2` uses that version -- No version: uses the single locked version, or errors if multiple exist - -**`(tool)` stanza**: - -- Version constraint in stanza: `(package (ocamlformat (= 0.26.2)))` -- Constraint applied when locking, not at runtime - -**Formatting rules** (`dune fmt`): - -1. Read `version` field from `.ocamlformat` file -2. If version present: use `%{tool:ocamlformat.}` pform -3. If no version: fall back to `%{bin:ocamlformat}` (system PATH) - -The pform handles locking/building/PATH fallback per the resolution rules above. - -#### Example scenarios - -| Scenario | Source | Version | -| ------------------------------------------- | -------------- | -------------- | -| `(tool (package ocamlformat))` in workspace | stanza | locked version | -| `dune tools add ocamlformat.0.26.2` | locked | 0.26.2 | -| `.ocamlformat` says `version=0.26.2` | locked or PATH | 0.26.2 | -| No config, ocamlformat on PATH | PATH | system version | - -CR-soon Alizter: Specify error messages for failure modes: solver fails, network -error, build fails, binary not found, multiple binaries without --bin, etc. Is -the locked version the most recent? - -### Directory structure - -_Implements: [1.2 Workspace-local](./requirements.md#12-workspace-local), -[1.5 Clean source tree](./requirements.md#15-clean-source-tree), -[3.2 Dependency isolation](./requirements.md#32-dependency-isolation)_ - -Tools use three types of directories: - -1. **External lock directories** - Created by `dune tools add`, contain solver - output -2. **Build lock directories** - Internal copies for build rules -3. **Install directories** - Built artifacts (binaries, libraries) - -``` -_build/ -├── .tools.lock/ # External lock directories (solver output) -│ └── / -│ └── / -│ ├── lock.dune # Lock file -│ └── .pkg # Package metadata -└── _private/ - └── default/ - ├── .tool-locks/ # Build lock directories (internal) - │ └── / - │ └── / - │ ├── lock.dune - │ └── .pkg - └── .tools/ # Install directories (built artifacts) - └── / - └── / - └── target/ - ├── bin/ # Installed executables - └── cookie # Installation trace (artifacts + variables) -``` - -**Path flow**: External lock → copied to build lock → built to install dir. - -**Key paths**: - -- External lock: `_build/.tools.lock///` -- Build lock: `_build/_private/default/.tool-locks///` -- Install: `_build/_private/default/.tools///target/` - -Note: The external lock directory (`_build/.tools.lock/`) is at the build root, -not inside `_private/`. This allows tools to be locked without going through the -build system. The build lock directory and install directory are both inside -`_private/default/` as they are managed by the build system. - -### Tool pforms TODO - -CR-Shon: I don't understand why this is necessary: why can't actions just keep -using the `bin` pform? Introduction of yet another configuration quirk should -have very good motive, but we already have a way to refer to executables, and -tools are just a way to manage the installation of executables. - -Maybe what we actually want is just way of specifying the scope of tool -versions? Perhaps via an extension to `env`: this seems like an `env` issue... - -_Implements: [5.4 Tool references in actions](#54-tool-references-in-actions)_ - -Build actions can reference tool executables via `%{tool:...}` pforms. - -#### Syntax - -``` -%{tool:} ; executable (default binary) -%{tool:.} ; specific version -``` - -#### Resolution - -1. If tool is locked, use the locked executable path -2. If tool is not locked, fall back to PATH (like `%{bin:...}`) - -CR-soon Alizter: What happens when version is omitted and multiple versions are -locked? Error, or pick one? - -#### Examples - -```lisp -;; Run a tool executable -(rule - (action (run %{tool:ocamlformat} %{deps}))) - -;; Use a specific version -(rule - (action (run %{tool:ocamlformat.0.26.2} %{deps}))) -``` - -#### Future: library access - -A future extension could allow accessing libraries from tool packages via -`%{tool:lib:}`. This would enable using a library solved independently from -the project's dependency graph. - -However, this has significant drawbacks: if two tool libraries share a common -dependency (e.g., both depend on `base`), they are solved independently and may -have incompatible versions. Using both in the same build rule could cause -linking errors. This needs further design work around "tool groups" or coherence -constraints before being viable. - -## Relationship to Package Management - -Tools are built on the same infrastructure as `dune pkg` (Dune's package -management): - -- Same solver (`opam-0install-solver`) -- Same repository system (opam-repository, overlays, custom repos) -- Same lock file format -- Same build rules for fetching and compiling packages - -### Orthogonality: a key design principle - -Tools are **orthogonal** to project package management. This decoupling -simplifies the design: - -- **No project lock required**: Tools work even without `dune.lock/`. A project - can use `dune tools` without enabling package management. -- **Independent solving**: Adding/removing tools doesn't affect project - dependencies, and vice versa. -- **No build invalidation**: Tool changes don't trigger project rebuilds - ([requirement 3.3](#33-no-build-triggers)). -- **Multi-version is natural**: Since each tool version is solved independently, - supporting multiple versions falls out of the design. - -The legacy system violated this principle by mixing tool dependencies with -project packages (the "wrapper package hack"), causing solver conflicts and -complex regeneration logic. - -### The package-tool continuum - -Rather than a binary choice between "package" and "tool", the design supports a -continuum of integration levels: - -**Fully integrated (packages)**: Dependencies in `dune.lock/` are solved -together, libraries and executables are available throughout the build, and -changes affect the entire project. - -**Fully isolated (tools)**: Each tool has its own lock directory, is solved -independently, and by default contributes nothing to the build environment. The -tool sits in `_build/` invisible to the project unless explicitly invoked. - -**Selective exposure (via pforms)**: The `%{tool:...}` pform -([Tool pforms](#tool-pforms)) allows build rules to selectively reference tool -executables. This lets a tool remain isolated (independently solved, no -dependency conflicts) while still being usable in specific rules. - -This continuum lets users choose the right level of integration: use packages -when you need tight coupling and shared dependencies; use tools when you want -isolation and version flexibility; use pforms when you need something in -between. - -### Key differences from project dependencies - -| Aspect | Project packages (`dune pkg`) | Tools (`dune tools`) | -| ------------- | --------------------------------- | ---------------------------------- | -| Lock location | `dune.lock/` (committed) | `_build/.tools.lock/` (ephemeral) | -| Solving | All packages solved together | Each tool solved independently | -| User workflow | `dune pkg lock` then `dune build` | `dune tools add`, builds on demand | -| Compiler | Project's compiler | Matches project or system OCaml | -| Requires lock | Yes | No (can use system OCaml) | - -CR-soon Alizter: system-ocaml, the opam package, is a bit of a hack. If a user -changes their opam switch it might not work as intended. - -CR-soon what does the Requires lock even mean here? - -### Why tools don't use `dune.lock/` - -Tools are stored in `_build/` rather than alongside `dune.lock/` because: - -1. **Tools are workspace infrastructure**, not project deliverables -2. **Different sub-projects may need different tool versions** (e.g., different - `.ocamlformat` versions per directory) -3. **Avoids cluttering source tree** (addresses [#10955], [#12097]) - -[#10955]: https://github.com/ocaml/dune/issues/10955 -[#12097]: https://github.com/ocaml/dune/issues/12097 - -The tradeoff is that tool lock directories are not committed to version control. -Reproducibility comes from `(tool)` stanzas in `dune-workspace`, which declare -version constraints that are re-solved when `_build/` is deleted. - -### Locking vs building - -These are separate phases, though `dune tools add` combines them: - -1. **Locking**: Solve dependencies, write lock directory with package metadata -2. **Building**: Download sources, compile tool, install to target directory - -Building (which includes downloading sources) happens lazily on first -`dune tools run`, not during `dune tools add`. The `add` command only solves -dependencies and writes the lock directory. - -**Open question**: How can users download, build, and install tools without -running them? This is useful for CI (prepare all tools upfront) or offline -scenarios. Options include a `--build` flag on `add`, a separate -`dune tools build` command, or relying on `dune tools path` to trigger builds. - -### Future: Relocatable compiler and caching - -Currently, each workspace builds its own copy of tools, even if the same tool -version is used across multiple projects. This is wasteful. - -The **relocatable compiler** work will enable: - -- **Shared build cache**: Tool builds can be cached and reused across projects -- **Faster tool installation**: If another project already built - `ocamlformat.0.26.2` with the same compiler, reuse the cached artifacts -- **Binary distribution**: Pre-built tool binaries could be distributed via - binary package repositories - -This will address the performance concerns of -[non-requirement 6.1](#61-system-wide-installation) (System-wide installation) -while maintaining project isolation. - -### Potential future stanza options - -The `(lock_dir)` stanza for project packages supports options not yet available -for tools: - -- `(solver_env ...)` - inject variables into solver -- `(constraints ...)` - additional solver constraints -- `(version_preference newest|oldest)` - prefer newest or oldest versions -- `(pins ...)` - pin packages to specific sources - -These may be added to `(tool)` stanzas if use cases emerge. In particular, -`(pins ...)` would help with [non-requirement 6.3](#63-tools-for-tools) (Tools -for tools) by allowing tools to be pinned to local paths. - -## GitHub Issues Addressed - -This section maps known issues to the requirements that address them. - -### Arbitrary tool packages ([#12913]) - -**Problem**: The current design requires every tool to be hardcoded. Only a few -tools are supported, and adding new ones requires code changes to dune. - -**Addressed by**: Requirement 1.1 (Generality). Any opam package that provides -an executable can be installed as a tool via `dune tools add `. - -[#12913]: https://github.com/ocaml/dune/issues/12913 - -### Lock directory clutter ([#10955], [#12097]) - -**Problem**: Users complained about lock directories cluttering the source tree. - -**Addressed by**: Requirement 1.5 (Clean source tree). Tool lock directories are -now stored in `_build/.tools.lock/`, not the source tree. - -[#10955]: https://github.com/ocaml/dune/issues/10955 -[#12097]: https://github.com/ocaml/dune/issues/12097 - -### Compiler ABI mismatch ([#11229]) - -**Problem**: Pre-built tool binaries (musl toolchain) don't work with -locally-built OCaml compilers because interface hashes don't match. - -**Addressed by**: Requirement 3.1 (Compiler compatibility). Tools are built from -source with matching compiler constraints by default. - -[#11229]: https://github.com/ocaml/dune/issues/11229 - -### Graceful fallback for missing tools ([#10578]) - -**Problem**: `dune fmt` fails hard when ocamlformat isn't installed, even if -other formatters could run. - -**Addressed by**: Requirements 2.4 (System PATH fallback) and 5.1 (Format -rules). When no version is specified in `.ocamlformat`, dune falls back to PATH. -If not found, the OCaml formatting rules are skipped rather than failing the -entire build. - -[#10578]: https://github.com/ocaml/dune/issues/10578 - -### Version constraints ([#12777]) - -**Problem**: No robust way to specify version constraints for dev tools. Users -had to hardcode paths to lock directories. - -**Addressed by**: Requirements 1.3 (Version pinning) and 4.2 (Persistent -configuration). The `(tool)` stanza allows declarative version constraints: - -```lisp -(tool - (package - (ocamlformat - (= 0.26.2)))) -``` - -[#12777]: https://github.com/ocaml/dune/issues/12777 - -### Version consistency ([#10688]) - -**Problem**: When ocamlformat isn't a project dependency, `dune fmt` captures -whatever version is on PATH, leading to inconsistent formatting. - -**Addressed by**: Requirements 1.3 (Version pinning) and 5.1 (Format rules). -When `.ocamlformat` specifies a version, dune uses the locked version. The PATH -is only used as fallback when no version is specified. - -[#10688]: https://github.com/ocaml/dune/issues/10688 - -### Dev tools redesign ([#12914]) - -**Problem**: Meta-issue tracking the need for a complete redesign of dev tools. - -**Addressed by**: This entire design document. Key improvements: - -- Generality (1.1): Any package, not just hardcoded ones -- Clean source tree (1.5): Lock dirs in `_build/` -- Version pinning (1.3): Declarative constraints via stanzas -- Multi-version (1.4): Different projects can use different versions -- Compiler compatibility (3.1): Tools built with matching compiler - -[#12914]: https://github.com/ocaml/dune/issues/12914 - -### Tool dependency isolation ([#12551]) - -**Problem**: Tools and project packages conflict when they share dependencies. - -**Addressed by**: Requirements 3.2 (Dependency isolation) and 5.3 (REPL). Each -tool has its own lock directory and is solved independently from project -dependencies. - -[#12551]: https://github.com/ocaml/dune/issues/12551 - -### No project build triggers ([#11037]) - -**Problem**: Running `dune fmt` triggers a full build of project dependencies, -even though formatting only needs ocamlformat. - -**Addressed by**: Requirements 3.3 (No build triggers) and 5.1 (Format rules). -Tool operations are isolated from project builds. - -[#11037]: https://github.com/ocaml/dune/issues/11037 - -### Compiler-independent formatters ([#11038]) - -**Problem**: `dune fmt` fails if no OCaml compiler is available, even though -ocamlformat could be a standalone binary. - -**Addressed by**: Requirement 3.1 (Compiler compatibility). Tools match the -project or system compiler, but formatters can opt out via -`(skip_compiler_match)`. - -[#11038]: https://github.com/ocaml/dune/issues/11038 - -### Version from config files ([#5315]) - -**Problem**: The `version` field in `.ocamlformat` should determine which -ocamlformat version to use. Different projects within a workspace may have -different `.ocamlformat` files specifying different versions. - -**Addressed by**: Requirements 1.3 (Version pinning), 1.4 (Multi-version -support), and 5.1 (Format rules). Formatting rules resolve the correct -ocamlformat version based on each directory's `.ocamlformat` file, and multiple -versions can coexist in the workspace. - -[#5315]: https://github.com/ocaml/dune/issues/5315 - -### Version constraint parsing ([#12866]) - -**Problem**: Version constraints for dev tools don't work correctly. - -**Addressed by**: Requirement 1.3 (Version pinning). The new `(tool)` stanza -properly supports version constraints using dependency syntax. - -[#12866]: https://github.com/ocaml/dune/issues/12866 - -### Opam with-dev-setup integration ([#12135]) - -**Problem**: Projects declare development tools in opam files using the -`:with-dev-setup` filter, but dune tools doesn't integrate with this. Users must -manually discover and add each tool, or duplicate declarations in both opam -files and `(tool)` stanzas. - -**Addressed by**: Requirement 4.1 (CLI), future work. See "Batch operations" in -the Specification for the planned `--with-dev-setup` flag. - -[#12135]: https://github.com/ocaml/dune/issues/12135 - -### Helpful error messages ([#13235], [#12975]) - -**Problem**: When tools aren't installed, error messages don't guide users -toward the correct commands. - -**Addressed by**: Requirement 4.1 (CLI). Commands should provide helpful -suggestions when tools aren't installed. - -[#13235]: https://github.com/ocaml/dune/issues/13235 -[#12975]: https://github.com/ocaml/dune/issues/12975 - -### Batch installation ([#12557]) - -**Problem**: Installing multiple tools requires multiple commands. - -**Addressed by**: Requirement 4.1 (CLI). `dune tools add` accepts multiple -package arguments, and with no arguments locks all tools from `(tool)` stanzas. -See "Batch operations" in the Specification. - -[#12557]: https://github.com/ocaml/dune/issues/12557 - -### Failure isolation ([#12818]) - -**Problem**: Failed tool installation can leave the environment in a broken -state. - -**Addressed by**: Requirement 3.2 (Dependency isolation). Each tool is isolated, -so a failed install doesn't affect other tools or the project. - -[#12818]: https://github.com/ocaml/dune/issues/12818 - -### Utop findlib integration ([#13471]) - -**Problem**: Running `#require` in utop triggers findlib warnings about -non-existent directories from dune's sandboxed package paths. - -**Addressed by**: Requirement 5.3 (REPL). When utop is configured as a tool, it -should be properly integrated with dune's package environment. - -[#13471]: https://github.com/ocaml/dune/issues/13471 - -## Comparison with Other Tools - -This section documents how other ecosystems handle tool management, informing -our design decisions and identifying gaps in our approach. - -### uv (Python) - -[uv][uv docs] distinguishes between ephemeral and persistent tool usage: - -- **`uvx `**: Runs tool in temporary virtual environment, cached but - disposable -- **`uv tool install `**: Persistent installation, executables on PATH - -**Key features**: - -- Per-tool isolated virtual environments (no cross-tool conflicts) -- Version syntax: `uvx ruff==0.3.0` (pinned), `uvx ruff@latest` (latest only) -- Constraint preservation: `uv tool install black>=23,<24` is respected by - `uv tool upgrade` - -**Relevance to requirements**: - -- 2.1: uv support adding its binary directory to shells https://docs.astral.sh/uv/concepts/tools/#tool-executables -- 2.2: - - uv uses a simple, predictable location for binary installations - https://docs.astral.sh/uv/reference/storage/#tool-executables and it allows - - uv allows configuration of the installation path and an API for locating the - bin directory https://docs.astral.sh/uv/reference/storage/#tool-executables -- **3.2 Dependency isolation**: Per-tool venvs achieve isolation. Our separate - lock dirs provide similar isolation. - -**Gap**: No project-local tool declaration. Tools are always user-global. This -violates our [requirement 1.2](#12-workspace-local) (Workspace-local). - -**What we should adopt**: - -- `upgrade` command that respects original constraints -- Ephemeral vs persistent distinction - -CR-soon Alizter: `dune tools upgrade` is not in the CLI specification. Add it or -note as future work. - -### cargo (Rust) - -[cargo install][cargo docs] provides minimal built-in tool support: - -- All binaries installed to single `~/.cargo/bin/` directory -- No isolation between tools -- No multi-version support (new version overwrites old) -- No project-local tools - -**Key features**: - -- Version constraint syntax: `cargo install ripgrep@1.2.0`, `--version ~1.2` -- Smart reinstall: only rebuilds if version/features/profile changed -- `cargo install --list` shows all installed packages - -**Relevance to requirements**: - -- **1.2 Workspace-local**: Violated. Cargo's model is entirely system-wide. -- **3.2 Dependency isolation**: Violated. All tools share one bin directory. - -**Gap**: No project-local tools, which is why cargo-run-bin exists. - -### cargo-run-bin (Rust community tool) - -[cargo-run-bin] is the **closest analog to our design**. It fills cargo's gaps -with project-local tool management: - -```toml -[package.metadata.bin] -cargo-nextest = { version = "0.9.57", locked = true } -dprint = { version = "0.30.3" } -cargo-mobile2 = { version = "0.5.2", bins = ["cargo-android", "cargo-mobile"] } -``` - -**Key features**: - -- Project-local `.bin/` cache directory -- Declarative config in `Cargo.toml` metadata section -- `bins` array for multi-binary packages -- `locked` flag for reproducible dependency resolution -- `cargo bin --install` to install all configured tools at once - -**Relevance to requirements**: - -- **1.2 Workspace-local**: ✓ Achieves this via `.bin/` directory -- **4.2 Persistent configuration**: ✓ Uses `Cargo.toml` metadata - -**What we should adopt**: - -- `bins` field for explicit binary listing (vs auto-discovery) -- `--locked` flag for reproducibility -- **Batch install command** (`dune tools add`). Critical for CI - -### npm (JavaScript) - -[npm][npm docs] has the most mature approach for project-local tools: - -- Tools declared in `devDependencies` in `package.json` -- Binaries linked to `node_modules/.bin/` -- `npx` runs binaries, preferring local over global -- npm scripts automatically get `node_modules/.bin` in PATH -- `package-lock.json` provides exact version pinning - -**Relevance to requirements**: - -- **1.2 Workspace-local**: ✓ `node_modules/.bin/` is project-local -- **2.2 Programmatic use**: ✓ Single directory for all tool binaries -- **4.2 Persistent configuration**: ✓ `devDependencies` in package.json - -**What we should adopt**: - -- devDependencies pattern (tools as dev deps, committed to repo) -- Auto-PATH for scripts (tools available without explicit path) -- `npx` prefers local, falls back to fetch - -### How Our Design Compares - -**Note**: Dune entries below describe **design requirements**, not necessarily -fully implemented features. Each entry references the relevant requirement -(e.g., "Req 1.2") to enable verification against the specification. - -#### Tools compared - -**uv** (Python): Fast Python package manager with tool support. Tools are -installed globally with per-tool isolated virtual environments. Supports -ephemeral runs (`uvx`) and constraint-preserving upgrades. No project-local or -declarative configuration. ([uv docs]) - -**cargo** (Rust): Rust's built-in package manager. Minimal tool support. All -binaries go to `~/.cargo/bin/`, no isolation, no multi-version. Smart reinstall -detects version changes. Compiler matching via rustup's `rust-toolchain.toml`. -([cargo docs]) - -**cargo-run-bin** (Rust): Community tool filling cargo's gaps. Project-local -`.bin/` cache, declarative config in `Cargo.toml` metadata, batch install. -Closest analog to our design in the Rust ecosystem. ([cargo-run-bin]) - -**npm** (JavaScript): Most mature project-local approach. Tools in -`devDependencies`, binaries in `node_modules/.bin/`, `npx` for execution. -Semver-respecting upgrades, lock file for reproducibility. ([npm docs]) - -#### Comparison table - -| Feature | dune | uv | cargo | cargo-run-bin | npm | -| -------------------- | ---- | --- | ----- | ------------- | --- | -| Project-local | ✓ | ✗ | ✗ | ✓ | ✓ | -| Multi-version | ✓ | ✗ | ✗ | ✗ | ✓\* | -| Declarative config | ✓ | ✗ | ✗ | ✓ | ✓ | -| Compiler matching | ✓ | ✗ | ✓\* | ✗ | ✗ | -| PATH fallback | ✓ | ✗ | ✗ | ✗ | ✓ | -| Precompiled binaries | ✗ | ✓ | ✗ | ✓\* | ✓ | -| Tool isolation | ✓ | ✓ | ✗ | ✓ | ✓ | -| Ephemeral runs | ✗ | ✓ | ✗ | ✗ | ✓ | -| Batch install | ✓ | ✗ | ✗ | ✓ | ✓ | -| Binary selection | ✓ | ✓ | ✓ | ✓ | ✓ | -| Tool discovery | ✓ | ✓ | ✗ | ✗ | ✓ | -| Tool upgrade | ✗ | ✓ | ✓\* | ✗ | ✓ | -| System-wide install | ✗ | ✓ | ✓ | ✗ | ✓ | - -_Asterisks indicate caveats; see relevant subsection for details._ - -#### Project-local - -Tools are installed within the project directory, isolated from other projects -and the global system. - -**Dune** (Req 1.2, 1.5): Tools are workspace-local. Lock files stored in -`_build/.tools.lock///`, built artifacts in -`_build/_private/default/.tools///`. - -**uv**: ✗ Tools installed globally. "Each tool environment is linked to a -specific Python version." Multiple Python versions require separate installs -with `--python` flag. If the Python version is uninstalled, the tool breaks. -([uv docs]) - -**cargo**: ✗ All binaries installed to single `~/.cargo/bin/`, shared by all -projects. Only one version per tool; reinstalling overwrites. Different versions -require custom `CARGO_INSTALL_ROOT` per project. ([cargo docs]) - -**cargo-run-bin**: ✓ Project-local `.bin/` cache directory. "Designed to manage -your Rust binaries within the context of your project." ([cargo-run-bin]) - -**npm**: ✓ Binaries linked to `node_modules/.bin/` per project. "Executables -linked into `./node_modules/.bin`." ([npm docs]) - -#### Multi-version - -Multiple versions of the same tool can coexist and be used within a single -workspace. - -**Dune** (Req 1.4): Versioned paths `_build/.tools.lock///` allow -multiple versions. Version selection: if 1 version locked, use it; if multiple, -error unless specified (e.g., `.ocamlformat` version field selects for -formatting). - -**uv**: ✗ "Installing a tool with the same name will replace the existing tool." -Open feature request for per-directory pinning (GitHub #17813). ([uv docs]) - -**cargo**: ✗ "If the package is already installed, Cargo will reinstall it if -the installed version does not appear to be up-to-date." Only one version per -install root; use `--force` to explicitly switch versions. ([cargo docs]) - -**cargo-run-bin**: ✗ Single version per tool in `Cargo.toml` metadata. -([cargo-run-bin]) - -**npm**: ✓\* Nested `node_modules/` can contain different versions for -transitive dependencies. Package aliases allow explicit multi-version: -`npm install react15@npm:react@^15`. For CLI tools, `npx pkg@version` runs -specific versions. ([npm docs: folders]) - -#### Declarative config - -Tools can be declared in a configuration file, enabling reproducible setups. - -**Dune** (Req 4.2): `(tool)` stanza in `dune-workspace`: - -```lisp -(tool - (package - (ocamlformat - (= 0.26.2)))) -``` - -**uv**: ✗ No project-level tool configuration. Tools installed via CLI only. -([uv docs]) - -**cargo**: ✗ No built-in declarative tool configuration. Feature request was -explicitly rejected as "NOT_PLANNED" - Cargo team directed users to third-party -tools instead. ([cargo#5120]) - -**cargo-run-bin**: ✓ Declared in `Cargo.toml` metadata section: - -```toml -[package.metadata.bin] -cargo-nextest = { version = "0.9.57" } -``` - -([cargo-run-bin]) - -**npm**: ✓ Declared in `package.json` devDependencies: - -```json -{ "devDependencies": { "prettier": "^3.0.0" } } -``` - -([npm docs: package.json]) - -#### Compiler matching - -Tools are built with a compiler version matching the project to ensure ABI -compatibility. - -**Dune** (Req 3.1): Automatic by default. Tools match project compiler or system -OCaml. Opt-out via `(skip_compiler_match)`. - -**uv**: ✗ Tools ignore project-level Python version settings. "Tool environment -details: Each tool environment is linked to a specific Python version... but -will ignore non-global Python version requests like `.python-version` files and -the `requires-python` value from a `pyproject.toml`." ([uv docs]) - -**cargo**: ✓* Via rustup (not cargo itself): `rust-toolchain.toml` in the -*current directory* affects `cargo install`. Note: this uses *your project's\* -toolchain, not the tool's - a known issue ([rust-lang/cargo#11036]). ([rustup -docs]) - -[rust-lang/cargo#11036]: https://github.com/rust-lang/cargo/issues/11036 - -**cargo-run-bin**: ✗ Uses whatever Rust toolchain is active. - -**npm**: ✗ No automatic matching. Packages declare Node version via `engines` -field, but npm only warns by default (requires `.npmrc` `engine-strict=true` to -enforce). Native addons use ABI version matching via node-pre-gyp. ([npm docs: -package.json]) - -#### PATH fallback - -When a tool is not installed via the tool manager, fall back to system PATH. - -**Dune** (Req 2.4): Build rules (e.g., `dune fmt`) fall back to system PATH when -no tool is locked. Behavior for `dune tools run` is an open question (see -below). - -**uv**: ✗ Does not search system PATH for tools. - -**cargo**: ✗ `cargo install` manages `~/.cargo/bin/`, but individual commands -don't fall back to system PATH. - -**cargo-run-bin**: ✗ Only runs binaries from local `.bin/` cache. - -**npm**: ✓ npx checks `$PATH` and `node_modules/.bin` before downloading. Use -`--ignore-existing` to skip PATH lookup. (Note: "shell fallback" was a different -feature—`--shell-auto-fallback`—removed in npm v7.) ([npm v7 changes]) - -#### Precompiled binaries - -Tools can be downloaded as precompiled binaries rather than built from source. - -**Dune** (Non-req 6.5): ✗ Tools always built from source. Ensures compiler -compatibility and avoids ABI issues. - -**uv**: ✓ Downloads prebuilt wheels when available. "uv downloads and installs -packages from PyPI." ([uv docs]) - -**cargo**: ✗ `cargo install` compiles from source. ([cargo docs]) - -**cargo-run-bin**: ✓\* Optional binstall support. "Download pre-compiled -binaries using cargo-binstall." ([cargo-run-bin]) - -**npm**: ✓ Packages are prebuilt JavaScript. Native addons may require -compilation. - -#### Tool isolation - -Each tool's dependencies are isolated from other tools and from project -dependencies. - -**Dune** (Req 3.2): Each tool solved independently with its own lock directory -in `_build/.tools.lock///`. - -**uv**: ✓ "Each tool has its own isolated virtual environment." ([uv docs]) - -**cargo**: ✗ All tools share single `~/.cargo/bin/` directory. No dependency -isolation between tools. - -**cargo-run-bin**: ✓ Each tool installed to separate directory under `.bin/`. -([cargo-run-bin]) - -**npm**: ✓ Each project has isolated `node_modules/`. Tools share dependencies -within a project but are isolated between projects. - -#### Ephemeral runs - -Run a tool once without persisting its installation - useful for trying tools or -running one-off commands without polluting the workspace's tool configuration. - -**Dune** (Non-req 6.6): ✗ Tools must be locked before running. Building from -source is slow, so caching (locking) is more practical than discarding. Users -can `dune tools remove` after if needed. - -**uv**: ✓ `uvx` runs in temporary environment. "Runs a tool without installing -it... The tool environment is cached." ([uv docs: uvx]) - -**cargo**: ✗ No ephemeral execution; `cargo install` always persists. - -**cargo-run-bin**: ✗ Tools must be configured and installed first. - -**npm**: ✓ `npx` can fetch and run without adding to package.json. "Run a -command from a local or remote npm package." ([npx docs]) - -#### Batch install - -Install all declared tools with a single command. - -**Dune** (Req 4.1): `dune tools add` with no arguments locks all tools from -`(tool)` stanzas. - -**uv**: ✗ No batch install; tools installed individually via `uv tool install`. - -**cargo**: ✗ No declarative tool configuration, so no batch install. - -**cargo-run-bin**: ✓ `cargo bin --install` installs all tools from config. -"cargo bin --install - Installs all binaries." ([cargo-run-bin]) - -**npm**: ✓ `npm install` installs all devDependencies. "Install all modules -listed as dependencies in package.json." ([npm docs]) - -#### Binary selection - -When a package provides multiple executables, how to specify which one to run. - -**Dune** (Req 1.6): `--bin` flag on `dune tools run` and `dune tools path`. -Auto-selects if package provides single binary; errors with helpful message if -multiple binaries exist. Declarative via `(executable ...)` in `(tool)` stanza. - -**uv**: ✓ `--from` option specifies package when command differs from package -name: `uvx --from httpie http`. Without `--from`, uvx assumes command name -equals package name. `uv tool install` installs all binaries from a package. -([uv docs]) - -**cargo**: ✓ `--bin NAME` flag selects specific binary. Installs all binaries by -default. `default-run` in Cargo.toml only affects `cargo run`, not install. -Binary name conflicts between packages cause errors. ([cargo docs]) - -**cargo-run-bin**: ✓ `bins` array in config specifies which binaries to build: -`cargo-mobile2 = { version = "0.5.2", bins = ["cargo-android", "cargo-mobile"] }`. -Builds all binaries if `bins` omitted. ([cargo-run-bin]) - -**npm**: ✓ `--package` (or `-p`) specifies package when binary name differs: -`npx --package=foo bar`. npx auto-matches binary to package name; errors if no -match. `bin` field in package.json maps command names to scripts. ([npx docs]) - -#### Tool discovery - -How scripts and editors programmatically find paths to installed tool -executables. - -**Dune** (Req 2.2): `dune tools path ` returns the executable path for a -specific tool. Per-tool granularity enables editors to find exact binaries. - -**uv**: ✓ `uv tool dir --bin` returns the bin directory (e.g., `~/.local/bin`). -`uv tool list` shows installed tools. No per-tool path command; construct path -as `$(uv tool dir --bin)/`. ([uv docs]) - -**cargo**: ✗ No discovery command. Binaries in `~/.cargo/bin/` (or -`$CARGO_INSTALL_ROOT/bin`). `cargo install --list` shows installed crates but -not paths. Scripts use `which ` or hardcode `~/.cargo/bin/`. ([cargo -docs]) - -**cargo-run-bin**: ✗ Binaries cached in `.bin/` directory. No CLI path command; -library API (`binary::install()`) returns paths programmatically. -([cargo-run-bin]) - -**npm**: ✓ `npm bin` returns `node_modules/.bin` directory (removed in npm 9+). -npm scripts automatically get `.bin` in PATH. No per-tool path command; use -`$(npm bin)/` or `./node_modules/.bin/`. ([npm docs: bin]) - -#### Tool upgrade - -Update installed tools to newer versions, optionally preserving version -constraints. - -**Dune** (Non-req 6.2): ✗ No upgrade command. Re-run `dune tools add ` to -get latest version, or `dune tools add .` for specific version. -Constraints from `(tool)` stanza preserved on re-lock. - -**uv**: ✓ `uv tool upgrade ` or `--all`. Respects original constraints: if -installed with `black>=23,<24`, upgrade stays within that range. To change -constraints, must reinstall with `uv tool install`. ([uv docs]) - -**cargo**: ✓\* `cargo install ` auto-detects newer versions without -`--force`. No constraint preservation; always installs latest (or specified) -version. Third-party `cargo-update` provides `cargo install-update -a` for batch -upgrades. ([cargo docs]) - -**cargo-run-bin**: ✗ No upgrade command. Edit version in `Cargo.toml` manually, -then run `cargo bin --install`. Version-controlled config ensures team sync. -([cargo-run-bin]) - -**npm**: ✓ `npm update` upgrades within semver constraints (e.g., `^1.2.3` -allows minor/patch updates). `npm outdated` shows available updates. For major -version bumps beyond constraints, use third-party `npm-check-updates`. ([npm -docs: update]) - -#### System-wide install - -Install tools globally, available outside any specific project. - -**Dune**: ✗ Intentionally omitted. Tools are workspace-local only. (See -[Non-requirement 6.1](#61-system-wide-installation)) - -**uv**: ✓ `uv tool install` adds to PATH. "Executables are installed to the uv -tools directory." ([uv docs]) - -**cargo**: ✓ `cargo install` puts binaries in `~/.cargo/bin/`, typically on -PATH. ([cargo docs]) - -**cargo-run-bin**: ✗ Project-local only. - -**npm**: ✓ `npm install -g` installs globally to system. ([npm docs]) - -### Recommendations from Research - -CR-soon Alizter: Review the entire Comparison with Other Tools section for -accuracy and completeness. Verify citations are still valid and that the -comparison table reflects current design decisions. - -**Adopted in this design**: - -- **Batch install** (from cargo-run-bin, npm): `dune tools add` with no - arguments locks all tools from `(tool)` stanzas (Req 4.1) -- **Declarative config** (from npm, cargo-run-bin): `(tool)` stanza in - `dune-workspace` (Req 4.2) -- **Binary selection** (from cargo-run-bin): `(executable ...)` field and - `--bin` flag (Req 1.6) - -**Future consideration**: - -- **`--locked` flag** (from cargo-run-bin): Ensure exact versions from solver, - not just constraints -- **Ephemeral run mode** (from uv's `uvx`): See - [Non-requirement 6.6](#66-ephemeral-runs) -- **System-wide linking** (from uv): See - [Non-requirement 6.1](#61-system-wide-installation) - -[uv docs]: https://docs.astral.sh/uv/concepts/tools/ -[uv docs: uvx]: https://docs.astral.sh/uv/concepts/tools/#running-tools -[cargo docs]: https://doc.rust-lang.org/cargo/commands/cargo-install.html -[cargo#5120]: https://github.com/rust-lang/cargo/issues/5120 -[rustup docs]: https://rust-lang.github.io/rustup/overrides.html -[cargo-run-bin]: https://github.com/dustinblackman/cargo-run-bin -[npm docs]: https://docs.npmjs.com/cli/ -[npm docs: folders]: https://docs.npmjs.com/cli/v9/configuring-npm/folders -[npm docs: package.json]: - https://docs.npmjs.com/cli/v10/configuring-npm/package-json -[npm docs: bin]: https://docs.npmjs.com/cli/v8/commands/npm-bin -[npm docs: update]: https://docs.npmjs.com/cli/v10/commands/npm-update -[npx docs]: https://docs.npmjs.com/cli/v10/commands/npx -[npm v7 changes]: - https://blog.npmjs.org/post/626173315965468672/npm-v7-series-beta-release-and-semver-major.html - -## Learnings from the Prototype - -CR-soon Alizter: This entire section may be considered garbage. Extract -important information out into the spec or remove entirely. - -This section documents key insights from the initial prototype implementation. - -### What Worked Well - -1. **Versioned lock directories**: Storing tools in - `_build/.tools.lock///` cleanly separates versions and keeps - the source tree clean. The flow (solve → extract version → move to versioned - path) is straightforward. - -2. **Tool isolation via synthetic wrapper**: Solving tools with a synthetic - `_tool_wrapper` package (not mixed with project packages) prevents - "packages outside workspace" solver errors. - -3. **Install cookie as build target**: Depending on `target/cookie` rather than - individual binaries correctly handles directory targets. The cookie is - created after full installation, providing a reliable completion signal. - -4. **Disk reads for lock checks**: Reading lock directories directly from disk - (not via build system) avoids triggering expensive project builds during tool - commands. - -5. **Simple PATH-only environment**: Tools only need the bin directory in PATH, - not a full exported environment. This avoids expensive closure computations. - -6. **Unified resolution with fallback**: A single resolution chain (stanza → - locked → PATH) provides clear priority and graceful degradation. - -### Problems Discovered - -1. **Directory targets cannot be depended upon directly**: Naively depending on - `_build/.../target/bin/` fails. Solution: depend on cookie, then read - cookie to discover binaries. - -2. **Build system triggers on lock checks**: Calling `Lock_dir.get` triggered - full builds. Solution: use `Lock_dir.read_disk` for checking. - -CR-soon Alizter: Need to understand what this is about. - -3. **Checksum collection missed versioned paths**: Fetch rules initially only - collected unversioned paths. Solution: scan `.tools.lock/` for all versions. - -4. **`.pkg/` rules gated by project lock**: Initial design skipped rule - generation when no project lock existed, breaking tools-only projects. - Solution: always generate rules; create tools-only DB when needed. - -5. **Compiler matching requires explicit package**: Just constraining `ocaml` - version wasn't enough; solver picked `ocaml-base-compiler` instead of - `ocaml-system`. Solution: explicitly require `ocaml-system` when system - compiler is available. - -### Reference Packages for Testing - -The implementation should be tested against these packages. We don't -special-case them, but they represent common tool patterns and historic -dev-tools support. - -CR-soon Alizter: Vet this list. Are there other common tools to add? Any that -should be removed? Consider tools from opam-repository's most-installed -packages. - -**Historic dev-tools** (from `src/dune_pkg/dev_tool.ml`): - -| Package | Binary | Notes | -| ------------------ | ---------------- | --------------------------------- | -| `ocamlformat` | `ocamlformat` | Also has `ocamlformat-rpc` binary | -| `odoc` | `odoc` | Documentation generator | -| `ocaml-lsp-server` | `ocamllsp` | Binary name differs from package | -| `utop` | `utop` | REPL, needs project libraries | -| `earlybird` | `ocamlearlybird` | Debugger | -| `odig` | `odig` | Documentation viewer | -| `opam-publish` | `opam-publish` | Release tooling | -| `dune-release` | `dune-release` | Release tooling | -| `ocaml-index` | `ocaml-index` | Indexer for ocamllsp | -| `merlin` | `ocamlmerlin` | Editor support | - -CR-soon Alizter: These are not historic dev tools, these are dev tools that we -currently support and would like to continue to support. Linking this to the -migration requirement. They will serve as test cases sure, but they also serve -as platform roadmap integration etc. - -**Edge cases to verify**: - -- Package with single binary: auto-selection works -- Package with multiple binaries: error with helpful message -- Package name ≠ binary name: `--bin` or `(executable)` works -- Tool needing project context: `utop` with `#require` - -CR-soon Alizter: I don't like these verifications here, they are out of place - -## Ideas - -CR-someday Alizter: Consider supporting `(tool)` stanzas in `dune-project` files -for per-project tool configuration, similar to how `(pin)` exists in both -`dune-workspace` and `dune-project`. This would allow different projects within -a workspace to declare different tool requirements. Conflict resolution: -workspace wins over project, outer project wins over nested. Sibling projects -with conflicting declarations can coexist. Within project-scoped rules the -correct tool is unambiguous. CLI commands without project context (e.g., -`dune tools run` from workspace root) should error if declarations conflict. - -CR-someday Alizter: Multi-version scaling. Supporting N versions means N -separate solves, downloads, and builds. In a monorepo migrating ocamlformat -versions across 50 sub-projects, users could accumulate many versions. Consider: -(1) warning when version count exceeds threshold, (2) `dune tools gc` to remove -unused versions, (3) documenting expected steady-state (few versions, not many). - -### Editor integration - -Editors and IDEs must be able to: - -- Discover which tools are available in a workspace -- Find the path to tool executables for spawning -- Invoke tools with dune managing the process (alternative pattern) - -See [CLI commands](./implementation.md#cli-commands) for the discovery and invocation interfaces. - -**Open issues** (require consultation with editors team): - -- How editors discover which tools are available -- LSP server invocation pattern: path discovery (editor spawns process) vs - managed invocation (dune manages process). LSP servers need editor control of - stdin/stdout for JSON-RPC and process lifecycle (restart on crash). -- Single-file formatting: `dune fmt ` for editor integration ([#3244] - discusses this) - -[#3244]: https://github.com/ocaml/dune/issues/3244 - -- `dune fmt` over RPC for watch mode integration -- Integration with `.ocamlformat` version detection -- Recommended editor configuration patterns - -CR-someday Alizter: Path staleness problem. If an editor caches the result of -`dune tools path ocamllsp` and the user later updates their OCaml version, the -cached path may point to an ABI-incompatible binary. There's no mechanism to -notify editors that cached paths are invalid. Options: (1) editors always call -`dune tools path` fresh, (2) dune provides a staleness check, (3) watch mode -integration notifies editors of tool rebuilds. - From eabd39b2f445e54f648184ab5eb3f7182e7fdd77 Mon Sep 17 00:00:00 2001 From: Shon Feder Date: Sat, 18 Jul 2026 23:46:25 -0400 Subject: [PATCH 69/72] Revert "Remove implementation spec" This reverts commit 3ec903e479b1c1f4abeff01ec3c8f9a7a120c4f4. --- doc/dev/tools-design/implementation.md | 1581 ++++++++++++++++++++++++ 1 file changed, 1581 insertions(+) create mode 100644 doc/dev/tools-design/implementation.md diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md new file mode 100644 index 00000000000..9258a90fbda --- /dev/null +++ b/doc/dev/tools-design/implementation.md @@ -0,0 +1,1581 @@ +## Tools Implementation + +Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) + +## Summary + +This document specifies the implementatino of dune's tool management system. + + + + +**Table of Contents** + +- [Tools Implementation](#tools-implementation) +- [Summary](#summary) +- [Design](#design) + - [The `(tool)` stanza](#the-tool-stanza) + - [Syntax](#syntax) + - [Fields](#fields) + - [Compiler matching](#compiler-matching) + - [Examples](#examples) + - [Invalid tool selections](#invalid-tool-selections) + - [CLI commands](#cli-commands) + - [Version syntax](#version-syntax) + - [Batch operations](#batch-operations) + - [Open question: version separator](#open-question-version-separator) + - [Open question: `add` vs `install` naming](#open-question-add-vs-install-naming) + - [Open question: `dune tools run` PATH fallback](#open-question-dune-tools-run-path-fallback) + - [Tool resolution](#tool-resolution) + - [Source resolution](#source-resolution) + - [Version selection](#version-selection) + - [Example scenarios](#example-scenarios) + - [Directory structure](#directory-structure) + - [Tool pforms TODO](#tool-pforms-todo) + - [Syntax](#syntax-1) + - [Resolution](#resolution) + - [Examples](#examples-1) + - [Future: library access](#future-library-access) +- [Relationship to Package Management](#relationship-to-package-management) + - [Orthogonality: a key design principle](#orthogonality-a-key-design-principle) + - [The package-tool continuum](#the-package-tool-continuum) + - [Key differences from project dependencies](#key-differences-from-project-dependencies) + - [Why tools don't use `dune.lock/`](#why-tools-dont-use-dunelock) + - [Locking vs building](#locking-vs-building) + - [Future: Relocatable compiler and caching](#future-relocatable-compiler-and-caching) + - [Potential future stanza options](#potential-future-stanza-options) +- [GitHub Issues Addressed](#github-issues-addressed) + - [Arbitrary tool packages ([#12913])](#arbitrary-tool-packages-12913) + - [Lock directory clutter ([#10955], [#12097])](#lock-directory-clutter-10955-12097) + - [Compiler ABI mismatch ([#11229])](#compiler-abi-mismatch-11229) + - [Graceful fallback for missing tools ([#10578])](#graceful-fallback-for-missing-tools-10578) + - [Version constraints ([#12777])](#version-constraints-12777) + - [Version consistency ([#10688])](#version-consistency-10688) + - [Dev tools redesign ([#12914])](#dev-tools-redesign-12914) + - [Tool dependency isolation ([#12551])](#tool-dependency-isolation-12551) + - [No project build triggers ([#11037])](#no-project-build-triggers-11037) + - [Compiler-independent formatters ([#11038])](#compiler-independent-formatters-11038) + - [Version from config files ([#5315])](#version-from-config-files-5315) + - [Version constraint parsing ([#12866])](#version-constraint-parsing-12866) + - [Opam with-dev-setup integration ([#12135])](#opam-with-dev-setup-integration-12135) + - [Helpful error messages ([#13235], [#12975])](#helpful-error-messages-13235-12975) + - [Batch installation ([#12557])](#batch-installation-12557) + - [Failure isolation ([#12818])](#failure-isolation-12818) + - [Utop findlib integration ([#13471])](#utop-findlib-integration-13471) +- [Comparison with Other Tools](#comparison-with-other-tools) + - [uv (Python)](#uv-python) + - [cargo (Rust)](#cargo-rust) + - [cargo-run-bin (Rust community tool)](#cargo-run-bin-rust-community-tool) + - [npm (JavaScript)](#npm-javascript) + - [How Our Design Compares](#how-our-design-compares) + - [Tools compared](#tools-compared) + - [Comparison table](#comparison-table) + - [Project-local](#project-local) + - [Multi-version](#multi-version) + - [Declarative config](#declarative-config) + - [Compiler matching](#compiler-matching-1) + - [PATH fallback](#path-fallback) + - [Precompiled binaries](#precompiled-binaries) + - [Tool isolation](#tool-isolation) + - [Ephemeral runs](#ephemeral-runs) + - [Batch install](#batch-install) + - [Binary selection](#binary-selection) + - [Tool discovery](#tool-discovery) + - [Tool upgrade](#tool-upgrade) + - [System-wide install](#system-wide-install) + - [Recommendations from Research](#recommendations-from-research) +- [Learnings from the Prototype](#learnings-from-the-prototype) + - [What Worked Well](#what-worked-well) + - [Problems Discovered](#problems-discovered) + - [Reference Packages for Testing](#reference-packages-for-testing) +- [Ideas](#ideas) + - [Editor integration](#editor-integration) + + + +## How to Read This Document + +[Design](#design) details _how_ the system implements the requirements: +stanza syntax, CLI commands, version resolution algorithm, and directory +structure. Each section notes which requirements it implements. Open questions +are marked where decisions are pending. + +[Relationship to Package Management](#relationship-to-package-management) +explains the orthogonality principle and how tools differ from project +dependencies. + +[Comparison with Other Tools](#comparison-with-other-tools) analyzes how +`uv`,`cargo`, `cargo-run-bin`, and `npm` handle tool management, informing our +design decisions. + +## Design + +### The `(tool)` stanza + +_Implements: [1.3 Version specification](./requirements.md#14-version-specification), +[1.6 Binary selection](./requirements.md#16-binary-selection), +[3.1 Compiler compatibility](./requirements.md#31-compiler-compatibility), +[4.2 Persistent configuration](./requirements.md#42-persistent-configuration)_ + +Tools are declared in `dune-workspace` using the `(tool)` stanza. The stanza +uses the same dependency syntax as `(depends ...)` for the package field. + +#### Syntax + +```lisp +(tool + (package ) ; Required: package name with optional constraint + (executable ) ; Optional: binary name, defaults to package name + (skip_compiler_match) ; Optional: flag to disable compiler matching + (repositories ...)) ; Optional: restrict to specific repositories +``` + +CR-soon Alizter: missing pin + +CR-soon Alizter: Should share some of lock_dir stanza fields. Need to do a +comparison and explain why it is or is not a good idea. + +CR-soon Alizter: Edge cases for stanza parsing: + +- Duplicate stanzas for same package with different constraints: error, or + merge? +- Invalid package name syntax: error message? +- Stanza references non-existent repository: when to error (parse time vs solve + time)? + +CR-soon Sudha247: Capture constraints here? Related issue: +[dune#12777](https://github.com/ocaml/dune/issues/12777) + +#### Fields + +- **`(package )`** (required): The opam package providing the tool. Accepts + either a plain package name or a name with version constraint: + - Plain: `ocamlformat` + - With constraint: `(ocamlformat (= 0.26.2))` + - Version operators: `=`, `<>`, `<`, `>`, `<=`, `>=` + - Conjunctions: `(and (>= 0.25.0) (< 0.27.0))` + +- **`(executable )`** (optional): The binary to run. Defaults to the + package name. Required when the package provides multiple binaries. + +- **`(skip_compiler_match)`** (optional): When present, disables the default + compiler matching behavior. Use this for tools that don't need compiler + compatibility (e.g., formatters that only parse source text). + +CR-soon Alizter: The prototype uses `compiler_compatible` (opt-in) but this doc +says `skip_compiler_match` (opt-out). Reconcile naming and semantics. + +- **`(repositories )`** (optional): Restrict package resolution to the + named repositories. Useful for binary package repositories. + +CR-soon Alizter: The `(repositories)` field is parsed but verify that the +restriction is actually applied during solving in the prototype. + +CR-soon Alizter: Missing pins + +#### Compiler matching + +_Implements: [3.1 Compiler compatibility](./requirements.md#31-compiler-compatibility)_ + +Tools that read project build artifacts (e.g., ocamllsp reading `.cmi` files) +must be built with a compatible compiler to avoid ABI mismatches. By default, +tools are built with a compiler matching the environment. + +The compiler is detected in this order: + +1. **Project compiler** - if using dune package management with a `dune.lock/` +2. **System OCaml** - from opam switch or PATH +3. **Any compatible version** - if no compiler is detected + +When `(skip_compiler_match)` is set, the tool is solved without compiler +constraints, allowing it to use any OCaml version. + +CR-soon Alizter: Edge cases for compiler matching: + +- No compiler found (no opam switch, no system OCaml): error, or pick any? +- System OCaml version changes between `add` and `run`: rebuild, or warn? +- Tool requires newer OCaml than project: error message? + +#### Examples + +```lisp +;; Minimal: just the package name +(tool + (package ocamlformat)) + +;; With version constraint +(tool + (package + (ocamlformat + (= 0.26.2)))) + +;; With version range +(tool + (package + (ocamlformat + (and + (>= 0.25.0) + (< 0.27.0))))) + +;; Specifying the binary for multi-binary packages +(tool + (package menhir) + (executable menhirSdk)) + +;; Tool that doesn't need compiler matching +(tool + (package ocamlformat) + (skip_compiler_match)) + +;; Using a binary package repository +(tool + (package ocaml-lsp-server) + (repositories binary-packages)) +``` + +#### Invalid tool selections + +Some packages cannot be used as tools: + +- **Library-only packages**: Packages that provide no executables (e.g., `base`, + `lwt`) cannot be tools. Error at `dune tools add` time. + +CR-someday Alizter: The `%{tool:lib:...}` pform in [Tool pforms](#tool-pforms) +could enable library-only tool packages, but this needs further design work. + +CR-soon Alizter: How can we error at dune tools add time? We would only find out +after building the tool. + +- **Compiler packages**: Selecting `ocaml-base-compiler` or `ocaml-system` as a + tool is invalid. These conflict with the compiler matching logic. + +- **Dune itself**: Using `dune` as a tool is the "in and out" problem + ([non-requirement 6.3](#63-tools-for-tools)). Circular dependency issues make + this out of scope. + +CR-soon Alizter: The explanations for compiler packages and dune itself need +more thought. + +CR-soon Alizter: Specify exact error messages for these cases. + +### CLI commands + +_Implements: [1.4 Version specification](./requirements.md##14-version-specification), +[1.6 Binary selection](./requirements.md#16-binary-selection), +[2.3 Programmatic use](./requirements.md#23-programmatic-use), [4.1 CLI](./requirements.md#41-cli)_ + +CR-soon Alizter: The CLI commands are generally underspecified. Each command +needs detailed behavior documentation: what triggers downloads/builds, how +errors are reported, interaction with stanzas vs CLI-added tools, exit codes, +output format, etc. + +``` +dune tools List all locked tools (alias for list) +dune tools list List all locked tools and versions + +dune tools add Lock all tools from (tool) stanzas +dune tools add [.] ... Lock specific tool(s) + +dune tools run [.] [--bin ] [-- ] + Run a tool, passing arguments after -- + +dune tools path Print path to the managed binary directory (or directories) +dune tools path [.] [--bin ] Print path to tool executable + +dune tools remove [.] ... Remove a tool's lock directory +``` + +The `--bin` flag is required when a package provides multiple binaries and no +`(executable ...)` is specified in the `(tool)` stanza. + +CR-soon Alizter: Edge cases for `dune tools add`: + +- Package doesn't exist in repository: error message? +- Network unavailable: error message, offline mode? +- Solver fails (unsatisfiable constraints): error with diagnostics? +- Build fails: partial state cleanup? +- Interrupted mid-install: cleanup, or resume on next run? +- Already locked at same version: no-op, or re-solve? +- Already locked at different version: add second version, or replace? + +CR-soon Alizter: The prototype has `--allow-not-installed` flag for +`dune tools path`. Document this flag and its use case (editor integration). + +CR-soon Alizter: `dune tools path` behavior is underspecified: + +- Should `path` trigger download and build if not yet built? Or only return path + if already built, erroring otherwise? +- If `--bin` is specified, can we compute the path without building (since path + is deterministic from package/version/binary name)? +- If package has multiple binaries and no `--bin`: error immediately, or + download and build first to discover available binaries? +- Tool not locked: error, or fall back to `which`? +- How does `path` interact with `(tool)` stanzas vs CLI-added tools? Same + resolution as `run`? + +CR-someday Alizter: Consider a single bin directory with symlinks to all tool +executables (like npm's `node_modules/.bin/`). Instead of per-tool paths, have +`_build/.tools/bin/` containing symlinks to all installed tool binaries. This +simplifies editor integration (one directory to add to PATH), avoids PATH length +limits (especially on Windows), and provides a single stable location for +discovery. Trade-off: need to maintain symlinks as tools are added/removed. +#### Version syntax + +The CLI uses dot-separated version syntax: `.` + +```bash +dune tools add ocamlformat # Latest version +dune tools add ocamlformat.0.26.2 # Specific version +``` + +#### Batch operations + +Batch operations apply to commands that manage tool state, not to commands that +use tools: + +| Command | Batch | Rationale | +| ------------------- | ----- | ----------------------------------------------- | +| `dune tools add` | ✓ | No args: all from stanzas; with args: specified | +| `dune tools remove` | ✓ | Multiple packages | +| `dune tools run` | ✗ | Running is inherently single-target | +| `dune tools path` | ✗ | Path lookup is per-tool | + +CR-soon Alizter: Edge cases for `dune tools remove`: + +- Tool not locked: error, or no-op with warning? +- Tool declared in stanza: remove lock but warn it will be re-added? +- Multiple versions locked: remove all, or require version specifier? +- Remove while tool is running: behavior? + +**`dune tools add`** has two modes: + +```bash +dune tools add # Lock all from (tool) stanzas +dune tools add ocamlformat odoc # Lock specific packages +dune tools add ocamlformat.0.26.2 odoc.2.4.0 # With version constraints +``` + +The no-argument form is critical for CI and onboarding - a single command to set +up all project tooling from declarative configuration. + +CR-someday Alizter: Integration with opam's `:with-dev-setup` marker ([#12135]): + +[#12135]: https://github.com/ocaml/dune/issues/12135 + +Projects often declare their development tools in opam files using the +`:with-dev-setup` filter (e.g., `ocamlformat :with-dev-setup`). Currently, dune +tools doesn't read these - users must manually add each tool or duplicate the +declarations in `(tool)` stanzas. + +One possible approach: + +```bash +dune tools add --with-dev-setup # Install tools from opam :with-dev-setup deps +dune tools add --with-doc # Install tools from :with-doc deps +``` + +This would enable: + +- Single source of truth for project tooling (opam file) +- Automatic tool discovery for new contributors +- CI scripts that don't need to know which tools to install + +CR-soon Alizter: Is `--with-dev-setup` the right approach? Alternatives: + +- Automatic: `dune tools add` (no args) always includes `:with-dev-setup` deps +- Separate command: `dune tools setup` specifically for opam marker integration +- No integration: keep `(tool)` stanzas as the only source, avoid duplication by + not using `:with-dev-setup` in opam files at all The flag approach adds + complexity. Need to evaluate if the use case justifies it. + +CR-soon Alizter: If we do integrate, design questions: + +- Read from `dune-project` `(depends)` or from `.opam` files? +- What if opam file says `ocamlformat` but `.ocamlformat` says `version=0.26.2`? +- How does this interact with `(tool)` stanzas that specify the same package? + +CR-soon Alizter: The prototype requires at least one package argument. The +no-argument batch mode (lock all from stanzas) is not yet implemented. + +#### Open question: version separator + +The current prototype uses `.` as the version separator (`pkg.version`), aligned +with opam conventions. However, `@` is common in other ecosystems: + +- **npm/npx**: `npx cowsay@1.5.0` +- **cargo**: `cargo install ripgrep@14.0.0` + +Opam package names cannot contain dots (allowed: `[a-zA-Z0-9_+-]+`), so `.` is +unambiguous. The choice is primarily about familiarity vs ecosystem consistency. + +#### Open question: `add` vs `install` naming + +The current prototype uses `dune tools add` for adding individual tools. +However, the naming convention is still open for discussion: + +- **Option A**: `add` for individual, `install` for batch (current prototype) +- **Option B**: `install` for both (consistent with legacy, npm, cargo) +- **Option C**: `lock` to emphasize the solving/locking aspect +- **Option D**: Semantic continuum where commands differ by how much work they + do: + - `add` = lock only (solve + write lock) + - `install` = lock + build (download + compile) + - `run` = lock + build + execute This gives users control over when work + happens (useful for CI, Docker builds). + +Considerations: + +- Legacy issues reference `dune tools install` +- `add` suggests incremental, `install` suggests setup +- `lock` aligns with `dune pkg lock` but may confuse (tools also build) +- Option D aligns with user expectations: "add to config" vs "install software" + +#### Open question: `dune tools run` PATH fallback + +Should `dune tools run ` fall back to system PATH when the tool is not +locked? + +- **Option A**: Fall back to PATH (consistent with build rules like `dune fmt`) +- **Option B**: Error with suggestion to lock or install on PATH (current impl) + +Considerations: + +- Falling back makes `dune tools run` a universal "run this tool" command +- Erroring is more explicit and avoids silently using unexpected versions +- Build rules need PATH fallback for gradual adoption (e.g., `dune fmt` without + locking ocamlformat) +- CLI commands could be stricter since the user explicitly requested a tool + +### Tool resolution + +_Implements: [1.5 Multi-version support](./requirements.md#15-multi-version-support), +[2.5 System PATH fallback](./requirements.md#25-system-path-fallback), +[5.1 Format rules](#51-format-rules)_ + +Tool resolution answers two questions: + +1. **Source**: Where does the tool come from? +2. **Version**: Which version of the tool? + +#### Source resolution + +When dune needs a tool, it checks these sources in order: + +1. **Workspace `(tool)` stanza** - declared in `dune-workspace` +2. **Locked versions** - previously added via `dune tools add` +3. **System PATH** - fallback if nothing else found + +The first source that has the tool wins. If no source has it, dune falls back to +system PATH (which may result in "command not found"). + +#### Version selection + +CR-soon Alizter: Version conflict resolution algorithm is undefined. What +happens when multiple sources specify conflicting versions? E.g., stanza says +`(= 0.26.2)` but `.ocamlformat` says `version=0.27.0`. Need to define precedence +or error behavior. + +How the version is determined depends on context: + +**CLI commands** (`dune tools run`, `dune tools add`): + +- Explicit version: `dune tools run ocamlformat.0.26.2` uses that version +- No version: uses the single locked version, or errors if multiple exist + +**`(tool)` stanza**: + +- Version constraint in stanza: `(package (ocamlformat (= 0.26.2)))` +- Constraint applied when locking, not at runtime + +**Formatting rules** (`dune fmt`): + +1. Read `version` field from `.ocamlformat` file +2. If version present: use `%{tool:ocamlformat.}` pform +3. If no version: fall back to `%{bin:ocamlformat}` (system PATH) + +The pform handles locking/building/PATH fallback per the resolution rules above. + +#### Example scenarios + +| Scenario | Source | Version | +| ------------------------------------------- | -------------- | -------------- | +| `(tool (package ocamlformat))` in workspace | stanza | locked version | +| `dune tools add ocamlformat.0.26.2` | locked | 0.26.2 | +| `.ocamlformat` says `version=0.26.2` | locked or PATH | 0.26.2 | +| No config, ocamlformat on PATH | PATH | system version | + +CR-soon Alizter: Specify error messages for failure modes: solver fails, network +error, build fails, binary not found, multiple binaries without --bin, etc. Is +the locked version the most recent? + +### Directory structure + +_Implements: [1.2 Workspace-local](./requirements.md#12-workspace-local), +[1.5 Clean source tree](./requirements.md#15-clean-source-tree), +[3.2 Dependency isolation](./requirements.md#32-dependency-isolation)_ + +Tools use three types of directories: + +1. **External lock directories** - Created by `dune tools add`, contain solver + output +2. **Build lock directories** - Internal copies for build rules +3. **Install directories** - Built artifacts (binaries, libraries) + +``` +_build/ +├── .tools.lock/ # External lock directories (solver output) +│ └── / +│ └── / +│ ├── lock.dune # Lock file +│ └── .pkg # Package metadata +└── _private/ + └── default/ + ├── .tool-locks/ # Build lock directories (internal) + │ └── / + │ └── / + │ ├── lock.dune + │ └── .pkg + └── .tools/ # Install directories (built artifacts) + └── / + └── / + └── target/ + ├── bin/ # Installed executables + └── cookie # Installation trace (artifacts + variables) +``` + +**Path flow**: External lock → copied to build lock → built to install dir. + +**Key paths**: + +- External lock: `_build/.tools.lock///` +- Build lock: `_build/_private/default/.tool-locks///` +- Install: `_build/_private/default/.tools///target/` + +Note: The external lock directory (`_build/.tools.lock/`) is at the build root, +not inside `_private/`. This allows tools to be locked without going through the +build system. The build lock directory and install directory are both inside +`_private/default/` as they are managed by the build system. + +### Tool pforms TODO + +CR-Shon: I don't understand why this is necessary: why can't actions just keep +using the `bin` pform? Introduction of yet another configuration quirk should +have very good motive, but we already have a way to refer to executables, and +tools are just a way to manage the installation of executables. + +Maybe what we actually want is just way of specifying the scope of tool +versions? Perhaps via an extension to `env`: this seems like an `env` issue... + +_Implements: [5.4 Tool references in actions](#54-tool-references-in-actions)_ + +Build actions can reference tool executables via `%{tool:...}` pforms. + +#### Syntax + +``` +%{tool:} ; executable (default binary) +%{tool:.} ; specific version +``` + +#### Resolution + +1. If tool is locked, use the locked executable path +2. If tool is not locked, fall back to PATH (like `%{bin:...}`) + +CR-soon Alizter: What happens when version is omitted and multiple versions are +locked? Error, or pick one? + +#### Examples + +```lisp +;; Run a tool executable +(rule + (action (run %{tool:ocamlformat} %{deps}))) + +;; Use a specific version +(rule + (action (run %{tool:ocamlformat.0.26.2} %{deps}))) +``` + +#### Future: library access + +A future extension could allow accessing libraries from tool packages via +`%{tool:lib:}`. This would enable using a library solved independently from +the project's dependency graph. + +However, this has significant drawbacks: if two tool libraries share a common +dependency (e.g., both depend on `base`), they are solved independently and may +have incompatible versions. Using both in the same build rule could cause +linking errors. This needs further design work around "tool groups" or coherence +constraints before being viable. + +## Relationship to Package Management + +Tools are built on the same infrastructure as `dune pkg` (Dune's package +management): + +- Same solver (`opam-0install-solver`) +- Same repository system (opam-repository, overlays, custom repos) +- Same lock file format +- Same build rules for fetching and compiling packages + +### Orthogonality: a key design principle + +Tools are **orthogonal** to project package management. This decoupling +simplifies the design: + +- **No project lock required**: Tools work even without `dune.lock/`. A project + can use `dune tools` without enabling package management. +- **Independent solving**: Adding/removing tools doesn't affect project + dependencies, and vice versa. +- **No build invalidation**: Tool changes don't trigger project rebuilds + ([requirement 3.3](#33-no-build-triggers)). +- **Multi-version is natural**: Since each tool version is solved independently, + supporting multiple versions falls out of the design. + +The legacy system violated this principle by mixing tool dependencies with +project packages (the "wrapper package hack"), causing solver conflicts and +complex regeneration logic. + +### The package-tool continuum + +Rather than a binary choice between "package" and "tool", the design supports a +continuum of integration levels: + +**Fully integrated (packages)**: Dependencies in `dune.lock/` are solved +together, libraries and executables are available throughout the build, and +changes affect the entire project. + +**Fully isolated (tools)**: Each tool has its own lock directory, is solved +independently, and by default contributes nothing to the build environment. The +tool sits in `_build/` invisible to the project unless explicitly invoked. + +**Selective exposure (via pforms)**: The `%{tool:...}` pform +([Tool pforms](#tool-pforms)) allows build rules to selectively reference tool +executables. This lets a tool remain isolated (independently solved, no +dependency conflicts) while still being usable in specific rules. + +This continuum lets users choose the right level of integration: use packages +when you need tight coupling and shared dependencies; use tools when you want +isolation and version flexibility; use pforms when you need something in +between. + +### Key differences from project dependencies + +| Aspect | Project packages (`dune pkg`) | Tools (`dune tools`) | +| ------------- | --------------------------------- | ---------------------------------- | +| Lock location | `dune.lock/` (committed) | `_build/.tools.lock/` (ephemeral) | +| Solving | All packages solved together | Each tool solved independently | +| User workflow | `dune pkg lock` then `dune build` | `dune tools add`, builds on demand | +| Compiler | Project's compiler | Matches project or system OCaml | +| Requires lock | Yes | No (can use system OCaml) | + +CR-soon Alizter: system-ocaml, the opam package, is a bit of a hack. If a user +changes their opam switch it might not work as intended. + +CR-soon what does the Requires lock even mean here? + +### Why tools don't use `dune.lock/` + +Tools are stored in `_build/` rather than alongside `dune.lock/` because: + +1. **Tools are workspace infrastructure**, not project deliverables +2. **Different sub-projects may need different tool versions** (e.g., different + `.ocamlformat` versions per directory) +3. **Avoids cluttering source tree** (addresses [#10955], [#12097]) + +[#10955]: https://github.com/ocaml/dune/issues/10955 +[#12097]: https://github.com/ocaml/dune/issues/12097 + +The tradeoff is that tool lock directories are not committed to version control. +Reproducibility comes from `(tool)` stanzas in `dune-workspace`, which declare +version constraints that are re-solved when `_build/` is deleted. + +### Locking vs building + +These are separate phases, though `dune tools add` combines them: + +1. **Locking**: Solve dependencies, write lock directory with package metadata +2. **Building**: Download sources, compile tool, install to target directory + +Building (which includes downloading sources) happens lazily on first +`dune tools run`, not during `dune tools add`. The `add` command only solves +dependencies and writes the lock directory. + +**Open question**: How can users download, build, and install tools without +running them? This is useful for CI (prepare all tools upfront) or offline +scenarios. Options include a `--build` flag on `add`, a separate +`dune tools build` command, or relying on `dune tools path` to trigger builds. + +### Future: Relocatable compiler and caching + +Currently, each workspace builds its own copy of tools, even if the same tool +version is used across multiple projects. This is wasteful. + +The **relocatable compiler** work will enable: + +- **Shared build cache**: Tool builds can be cached and reused across projects +- **Faster tool installation**: If another project already built + `ocamlformat.0.26.2` with the same compiler, reuse the cached artifacts +- **Binary distribution**: Pre-built tool binaries could be distributed via + binary package repositories + +This will address the performance concerns of +[non-requirement 6.1](#61-system-wide-installation) (System-wide installation) +while maintaining project isolation. + +### Potential future stanza options + +The `(lock_dir)` stanza for project packages supports options not yet available +for tools: + +- `(solver_env ...)` - inject variables into solver +- `(constraints ...)` - additional solver constraints +- `(version_preference newest|oldest)` - prefer newest or oldest versions +- `(pins ...)` - pin packages to specific sources + +These may be added to `(tool)` stanzas if use cases emerge. In particular, +`(pins ...)` would help with [non-requirement 6.3](#63-tools-for-tools) (Tools +for tools) by allowing tools to be pinned to local paths. + +## GitHub Issues Addressed + +This section maps known issues to the requirements that address them. + +### Arbitrary tool packages ([#12913]) + +**Problem**: The current design requires every tool to be hardcoded. Only a few +tools are supported, and adding new ones requires code changes to dune. + +**Addressed by**: Requirement 1.1 (Generality). Any opam package that provides +an executable can be installed as a tool via `dune tools add `. + +[#12913]: https://github.com/ocaml/dune/issues/12913 + +### Lock directory clutter ([#10955], [#12097]) + +**Problem**: Users complained about lock directories cluttering the source tree. + +**Addressed by**: Requirement 1.5 (Clean source tree). Tool lock directories are +now stored in `_build/.tools.lock/`, not the source tree. + +[#10955]: https://github.com/ocaml/dune/issues/10955 +[#12097]: https://github.com/ocaml/dune/issues/12097 + +### Compiler ABI mismatch ([#11229]) + +**Problem**: Pre-built tool binaries (musl toolchain) don't work with +locally-built OCaml compilers because interface hashes don't match. + +**Addressed by**: Requirement 3.1 (Compiler compatibility). Tools are built from +source with matching compiler constraints by default. + +[#11229]: https://github.com/ocaml/dune/issues/11229 + +### Graceful fallback for missing tools ([#10578]) + +**Problem**: `dune fmt` fails hard when ocamlformat isn't installed, even if +other formatters could run. + +**Addressed by**: Requirements 2.4 (System PATH fallback) and 5.1 (Format +rules). When no version is specified in `.ocamlformat`, dune falls back to PATH. +If not found, the OCaml formatting rules are skipped rather than failing the +entire build. + +[#10578]: https://github.com/ocaml/dune/issues/10578 + +### Version constraints ([#12777]) + +**Problem**: No robust way to specify version constraints for dev tools. Users +had to hardcode paths to lock directories. + +**Addressed by**: Requirements 1.3 (Version pinning) and 4.2 (Persistent +configuration). The `(tool)` stanza allows declarative version constraints: + +```lisp +(tool + (package + (ocamlformat + (= 0.26.2)))) +``` + +[#12777]: https://github.com/ocaml/dune/issues/12777 + +### Version consistency ([#10688]) + +**Problem**: When ocamlformat isn't a project dependency, `dune fmt` captures +whatever version is on PATH, leading to inconsistent formatting. + +**Addressed by**: Requirements 1.3 (Version pinning) and 5.1 (Format rules). +When `.ocamlformat` specifies a version, dune uses the locked version. The PATH +is only used as fallback when no version is specified. + +[#10688]: https://github.com/ocaml/dune/issues/10688 + +### Dev tools redesign ([#12914]) + +**Problem**: Meta-issue tracking the need for a complete redesign of dev tools. + +**Addressed by**: This entire design document. Key improvements: + +- Generality (1.1): Any package, not just hardcoded ones +- Clean source tree (1.5): Lock dirs in `_build/` +- Version pinning (1.3): Declarative constraints via stanzas +- Multi-version (1.4): Different projects can use different versions +- Compiler compatibility (3.1): Tools built with matching compiler + +[#12914]: https://github.com/ocaml/dune/issues/12914 + +### Tool dependency isolation ([#12551]) + +**Problem**: Tools and project packages conflict when they share dependencies. + +**Addressed by**: Requirements 3.2 (Dependency isolation) and 5.3 (REPL). Each +tool has its own lock directory and is solved independently from project +dependencies. + +[#12551]: https://github.com/ocaml/dune/issues/12551 + +### No project build triggers ([#11037]) + +**Problem**: Running `dune fmt` triggers a full build of project dependencies, +even though formatting only needs ocamlformat. + +**Addressed by**: Requirements 3.3 (No build triggers) and 5.1 (Format rules). +Tool operations are isolated from project builds. + +[#11037]: https://github.com/ocaml/dune/issues/11037 + +### Compiler-independent formatters ([#11038]) + +**Problem**: `dune fmt` fails if no OCaml compiler is available, even though +ocamlformat could be a standalone binary. + +**Addressed by**: Requirement 3.1 (Compiler compatibility). Tools match the +project or system compiler, but formatters can opt out via +`(skip_compiler_match)`. + +[#11038]: https://github.com/ocaml/dune/issues/11038 + +### Version from config files ([#5315]) + +**Problem**: The `version` field in `.ocamlformat` should determine which +ocamlformat version to use. Different projects within a workspace may have +different `.ocamlformat` files specifying different versions. + +**Addressed by**: Requirements 1.3 (Version pinning), 1.4 (Multi-version +support), and 5.1 (Format rules). Formatting rules resolve the correct +ocamlformat version based on each directory's `.ocamlformat` file, and multiple +versions can coexist in the workspace. + +[#5315]: https://github.com/ocaml/dune/issues/5315 + +### Version constraint parsing ([#12866]) + +**Problem**: Version constraints for dev tools don't work correctly. + +**Addressed by**: Requirement 1.3 (Version pinning). The new `(tool)` stanza +properly supports version constraints using dependency syntax. + +[#12866]: https://github.com/ocaml/dune/issues/12866 + +### Opam with-dev-setup integration ([#12135]) + +**Problem**: Projects declare development tools in opam files using the +`:with-dev-setup` filter, but dune tools doesn't integrate with this. Users must +manually discover and add each tool, or duplicate declarations in both opam +files and `(tool)` stanzas. + +**Addressed by**: Requirement 4.1 (CLI), future work. See "Batch operations" in +the Specification for the planned `--with-dev-setup` flag. + +[#12135]: https://github.com/ocaml/dune/issues/12135 + +### Helpful error messages ([#13235], [#12975]) + +**Problem**: When tools aren't installed, error messages don't guide users +toward the correct commands. + +**Addressed by**: Requirement 4.1 (CLI). Commands should provide helpful +suggestions when tools aren't installed. + +[#13235]: https://github.com/ocaml/dune/issues/13235 +[#12975]: https://github.com/ocaml/dune/issues/12975 + +### Batch installation ([#12557]) + +**Problem**: Installing multiple tools requires multiple commands. + +**Addressed by**: Requirement 4.1 (CLI). `dune tools add` accepts multiple +package arguments, and with no arguments locks all tools from `(tool)` stanzas. +See "Batch operations" in the Specification. + +[#12557]: https://github.com/ocaml/dune/issues/12557 + +### Failure isolation ([#12818]) + +**Problem**: Failed tool installation can leave the environment in a broken +state. + +**Addressed by**: Requirement 3.2 (Dependency isolation). Each tool is isolated, +so a failed install doesn't affect other tools or the project. + +[#12818]: https://github.com/ocaml/dune/issues/12818 + +### Utop findlib integration ([#13471]) + +**Problem**: Running `#require` in utop triggers findlib warnings about +non-existent directories from dune's sandboxed package paths. + +**Addressed by**: Requirement 5.3 (REPL). When utop is configured as a tool, it +should be properly integrated with dune's package environment. + +[#13471]: https://github.com/ocaml/dune/issues/13471 + +## Comparison with Other Tools + +This section documents how other ecosystems handle tool management, informing +our design decisions and identifying gaps in our approach. + +### uv (Python) + +[uv][uv docs] distinguishes between ephemeral and persistent tool usage: + +- **`uvx `**: Runs tool in temporary virtual environment, cached but + disposable +- **`uv tool install `**: Persistent installation, executables on PATH + +**Key features**: + +- Per-tool isolated virtual environments (no cross-tool conflicts) +- Version syntax: `uvx ruff==0.3.0` (pinned), `uvx ruff@latest` (latest only) +- Constraint preservation: `uv tool install black>=23,<24` is respected by + `uv tool upgrade` + +**Relevance to requirements**: + +- 2.1: uv support adding its binary directory to shells https://docs.astral.sh/uv/concepts/tools/#tool-executables +- 2.2: + - uv uses a simple, predictable location for binary installations + https://docs.astral.sh/uv/reference/storage/#tool-executables and it allows + - uv allows configuration of the installation path and an API for locating the + bin directory https://docs.astral.sh/uv/reference/storage/#tool-executables +- **3.2 Dependency isolation**: Per-tool venvs achieve isolation. Our separate + lock dirs provide similar isolation. + +**Gap**: No project-local tool declaration. Tools are always user-global. This +violates our [requirement 1.2](#12-workspace-local) (Workspace-local). + +**What we should adopt**: + +- `upgrade` command that respects original constraints +- Ephemeral vs persistent distinction + +CR-soon Alizter: `dune tools upgrade` is not in the CLI specification. Add it or +note as future work. + +### cargo (Rust) + +[cargo install][cargo docs] provides minimal built-in tool support: + +- All binaries installed to single `~/.cargo/bin/` directory +- No isolation between tools +- No multi-version support (new version overwrites old) +- No project-local tools + +**Key features**: + +- Version constraint syntax: `cargo install ripgrep@1.2.0`, `--version ~1.2` +- Smart reinstall: only rebuilds if version/features/profile changed +- `cargo install --list` shows all installed packages + +**Relevance to requirements**: + +- **1.2 Workspace-local**: Violated. Cargo's model is entirely system-wide. +- **3.2 Dependency isolation**: Violated. All tools share one bin directory. + +**Gap**: No project-local tools, which is why cargo-run-bin exists. + +### cargo-run-bin (Rust community tool) + +[cargo-run-bin] is the **closest analog to our design**. It fills cargo's gaps +with project-local tool management: + +```toml +[package.metadata.bin] +cargo-nextest = { version = "0.9.57", locked = true } +dprint = { version = "0.30.3" } +cargo-mobile2 = { version = "0.5.2", bins = ["cargo-android", "cargo-mobile"] } +``` + +**Key features**: + +- Project-local `.bin/` cache directory +- Declarative config in `Cargo.toml` metadata section +- `bins` array for multi-binary packages +- `locked` flag for reproducible dependency resolution +- `cargo bin --install` to install all configured tools at once + +**Relevance to requirements**: + +- **1.2 Workspace-local**: ✓ Achieves this via `.bin/` directory +- **4.2 Persistent configuration**: ✓ Uses `Cargo.toml` metadata + +**What we should adopt**: + +- `bins` field for explicit binary listing (vs auto-discovery) +- `--locked` flag for reproducibility +- **Batch install command** (`dune tools add`). Critical for CI + +### npm (JavaScript) + +[npm][npm docs] has the most mature approach for project-local tools: + +- Tools declared in `devDependencies` in `package.json` +- Binaries linked to `node_modules/.bin/` +- `npx` runs binaries, preferring local over global +- npm scripts automatically get `node_modules/.bin` in PATH +- `package-lock.json` provides exact version pinning + +**Relevance to requirements**: + +- **1.2 Workspace-local**: ✓ `node_modules/.bin/` is project-local +- **2.2 Programmatic use**: ✓ Single directory for all tool binaries +- **4.2 Persistent configuration**: ✓ `devDependencies` in package.json + +**What we should adopt**: + +- devDependencies pattern (tools as dev deps, committed to repo) +- Auto-PATH for scripts (tools available without explicit path) +- `npx` prefers local, falls back to fetch + +### How Our Design Compares + +**Note**: Dune entries below describe **design requirements**, not necessarily +fully implemented features. Each entry references the relevant requirement +(e.g., "Req 1.2") to enable verification against the specification. + +#### Tools compared + +**uv** (Python): Fast Python package manager with tool support. Tools are +installed globally with per-tool isolated virtual environments. Supports +ephemeral runs (`uvx`) and constraint-preserving upgrades. No project-local or +declarative configuration. ([uv docs]) + +**cargo** (Rust): Rust's built-in package manager. Minimal tool support. All +binaries go to `~/.cargo/bin/`, no isolation, no multi-version. Smart reinstall +detects version changes. Compiler matching via rustup's `rust-toolchain.toml`. +([cargo docs]) + +**cargo-run-bin** (Rust): Community tool filling cargo's gaps. Project-local +`.bin/` cache, declarative config in `Cargo.toml` metadata, batch install. +Closest analog to our design in the Rust ecosystem. ([cargo-run-bin]) + +**npm** (JavaScript): Most mature project-local approach. Tools in +`devDependencies`, binaries in `node_modules/.bin/`, `npx` for execution. +Semver-respecting upgrades, lock file for reproducibility. ([npm docs]) + +#### Comparison table + +| Feature | dune | uv | cargo | cargo-run-bin | npm | +| -------------------- | ---- | --- | ----- | ------------- | --- | +| Project-local | ✓ | ✗ | ✗ | ✓ | ✓ | +| Multi-version | ✓ | ✗ | ✗ | ✗ | ✓\* | +| Declarative config | ✓ | ✗ | ✗ | ✓ | ✓ | +| Compiler matching | ✓ | ✗ | ✓\* | ✗ | ✗ | +| PATH fallback | ✓ | ✗ | ✗ | ✗ | ✓ | +| Precompiled binaries | ✗ | ✓ | ✗ | ✓\* | ✓ | +| Tool isolation | ✓ | ✓ | ✗ | ✓ | ✓ | +| Ephemeral runs | ✗ | ✓ | ✗ | ✗ | ✓ | +| Batch install | ✓ | ✗ | ✗ | ✓ | ✓ | +| Binary selection | ✓ | ✓ | ✓ | ✓ | ✓ | +| Tool discovery | ✓ | ✓ | ✗ | ✗ | ✓ | +| Tool upgrade | ✗ | ✓ | ✓\* | ✗ | ✓ | +| System-wide install | ✗ | ✓ | ✓ | ✗ | ✓ | + +_Asterisks indicate caveats; see relevant subsection for details._ + +#### Project-local + +Tools are installed within the project directory, isolated from other projects +and the global system. + +**Dune** (Req 1.2, 1.5): Tools are workspace-local. Lock files stored in +`_build/.tools.lock///`, built artifacts in +`_build/_private/default/.tools///`. + +**uv**: ✗ Tools installed globally. "Each tool environment is linked to a +specific Python version." Multiple Python versions require separate installs +with `--python` flag. If the Python version is uninstalled, the tool breaks. +([uv docs]) + +**cargo**: ✗ All binaries installed to single `~/.cargo/bin/`, shared by all +projects. Only one version per tool; reinstalling overwrites. Different versions +require custom `CARGO_INSTALL_ROOT` per project. ([cargo docs]) + +**cargo-run-bin**: ✓ Project-local `.bin/` cache directory. "Designed to manage +your Rust binaries within the context of your project." ([cargo-run-bin]) + +**npm**: ✓ Binaries linked to `node_modules/.bin/` per project. "Executables +linked into `./node_modules/.bin`." ([npm docs]) + +#### Multi-version + +Multiple versions of the same tool can coexist and be used within a single +workspace. + +**Dune** (Req 1.4): Versioned paths `_build/.tools.lock///` allow +multiple versions. Version selection: if 1 version locked, use it; if multiple, +error unless specified (e.g., `.ocamlformat` version field selects for +formatting). + +**uv**: ✗ "Installing a tool with the same name will replace the existing tool." +Open feature request for per-directory pinning (GitHub #17813). ([uv docs]) + +**cargo**: ✗ "If the package is already installed, Cargo will reinstall it if +the installed version does not appear to be up-to-date." Only one version per +install root; use `--force` to explicitly switch versions. ([cargo docs]) + +**cargo-run-bin**: ✗ Single version per tool in `Cargo.toml` metadata. +([cargo-run-bin]) + +**npm**: ✓\* Nested `node_modules/` can contain different versions for +transitive dependencies. Package aliases allow explicit multi-version: +`npm install react15@npm:react@^15`. For CLI tools, `npx pkg@version` runs +specific versions. ([npm docs: folders]) + +#### Declarative config + +Tools can be declared in a configuration file, enabling reproducible setups. + +**Dune** (Req 4.2): `(tool)` stanza in `dune-workspace`: + +```lisp +(tool + (package + (ocamlformat + (= 0.26.2)))) +``` + +**uv**: ✗ No project-level tool configuration. Tools installed via CLI only. +([uv docs]) + +**cargo**: ✗ No built-in declarative tool configuration. Feature request was +explicitly rejected as "NOT_PLANNED" - Cargo team directed users to third-party +tools instead. ([cargo#5120]) + +**cargo-run-bin**: ✓ Declared in `Cargo.toml` metadata section: + +```toml +[package.metadata.bin] +cargo-nextest = { version = "0.9.57" } +``` + +([cargo-run-bin]) + +**npm**: ✓ Declared in `package.json` devDependencies: + +```json +{ "devDependencies": { "prettier": "^3.0.0" } } +``` + +([npm docs: package.json]) + +#### Compiler matching + +Tools are built with a compiler version matching the project to ensure ABI +compatibility. + +**Dune** (Req 3.1): Automatic by default. Tools match project compiler or system +OCaml. Opt-out via `(skip_compiler_match)`. + +**uv**: ✗ Tools ignore project-level Python version settings. "Tool environment +details: Each tool environment is linked to a specific Python version... but +will ignore non-global Python version requests like `.python-version` files and +the `requires-python` value from a `pyproject.toml`." ([uv docs]) + +**cargo**: ✓* Via rustup (not cargo itself): `rust-toolchain.toml` in the +*current directory* affects `cargo install`. Note: this uses *your project's\* +toolchain, not the tool's - a known issue ([rust-lang/cargo#11036]). ([rustup +docs]) + +[rust-lang/cargo#11036]: https://github.com/rust-lang/cargo/issues/11036 + +**cargo-run-bin**: ✗ Uses whatever Rust toolchain is active. + +**npm**: ✗ No automatic matching. Packages declare Node version via `engines` +field, but npm only warns by default (requires `.npmrc` `engine-strict=true` to +enforce). Native addons use ABI version matching via node-pre-gyp. ([npm docs: +package.json]) + +#### PATH fallback + +When a tool is not installed via the tool manager, fall back to system PATH. + +**Dune** (Req 2.4): Build rules (e.g., `dune fmt`) fall back to system PATH when +no tool is locked. Behavior for `dune tools run` is an open question (see +below). + +**uv**: ✗ Does not search system PATH for tools. + +**cargo**: ✗ `cargo install` manages `~/.cargo/bin/`, but individual commands +don't fall back to system PATH. + +**cargo-run-bin**: ✗ Only runs binaries from local `.bin/` cache. + +**npm**: ✓ npx checks `$PATH` and `node_modules/.bin` before downloading. Use +`--ignore-existing` to skip PATH lookup. (Note: "shell fallback" was a different +feature—`--shell-auto-fallback`—removed in npm v7.) ([npm v7 changes]) + +#### Precompiled binaries + +Tools can be downloaded as precompiled binaries rather than built from source. + +**Dune** (Non-req 6.5): ✗ Tools always built from source. Ensures compiler +compatibility and avoids ABI issues. + +**uv**: ✓ Downloads prebuilt wheels when available. "uv downloads and installs +packages from PyPI." ([uv docs]) + +**cargo**: ✗ `cargo install` compiles from source. ([cargo docs]) + +**cargo-run-bin**: ✓\* Optional binstall support. "Download pre-compiled +binaries using cargo-binstall." ([cargo-run-bin]) + +**npm**: ✓ Packages are prebuilt JavaScript. Native addons may require +compilation. + +#### Tool isolation + +Each tool's dependencies are isolated from other tools and from project +dependencies. + +**Dune** (Req 3.2): Each tool solved independently with its own lock directory +in `_build/.tools.lock///`. + +**uv**: ✓ "Each tool has its own isolated virtual environment." ([uv docs]) + +**cargo**: ✗ All tools share single `~/.cargo/bin/` directory. No dependency +isolation between tools. + +**cargo-run-bin**: ✓ Each tool installed to separate directory under `.bin/`. +([cargo-run-bin]) + +**npm**: ✓ Each project has isolated `node_modules/`. Tools share dependencies +within a project but are isolated between projects. + +#### Ephemeral runs + +Run a tool once without persisting its installation - useful for trying tools or +running one-off commands without polluting the workspace's tool configuration. + +**Dune** (Non-req 6.6): ✗ Tools must be locked before running. Building from +source is slow, so caching (locking) is more practical than discarding. Users +can `dune tools remove` after if needed. + +**uv**: ✓ `uvx` runs in temporary environment. "Runs a tool without installing +it... The tool environment is cached." ([uv docs: uvx]) + +**cargo**: ✗ No ephemeral execution; `cargo install` always persists. + +**cargo-run-bin**: ✗ Tools must be configured and installed first. + +**npm**: ✓ `npx` can fetch and run without adding to package.json. "Run a +command from a local or remote npm package." ([npx docs]) + +#### Batch install + +Install all declared tools with a single command. + +**Dune** (Req 4.1): `dune tools add` with no arguments locks all tools from +`(tool)` stanzas. + +**uv**: ✗ No batch install; tools installed individually via `uv tool install`. + +**cargo**: ✗ No declarative tool configuration, so no batch install. + +**cargo-run-bin**: ✓ `cargo bin --install` installs all tools from config. +"cargo bin --install - Installs all binaries." ([cargo-run-bin]) + +**npm**: ✓ `npm install` installs all devDependencies. "Install all modules +listed as dependencies in package.json." ([npm docs]) + +#### Binary selection + +When a package provides multiple executables, how to specify which one to run. + +**Dune** (Req 1.6): `--bin` flag on `dune tools run` and `dune tools path`. +Auto-selects if package provides single binary; errors with helpful message if +multiple binaries exist. Declarative via `(executable ...)` in `(tool)` stanza. + +**uv**: ✓ `--from` option specifies package when command differs from package +name: `uvx --from httpie http`. Without `--from`, uvx assumes command name +equals package name. `uv tool install` installs all binaries from a package. +([uv docs]) + +**cargo**: ✓ `--bin NAME` flag selects specific binary. Installs all binaries by +default. `default-run` in Cargo.toml only affects `cargo run`, not install. +Binary name conflicts between packages cause errors. ([cargo docs]) + +**cargo-run-bin**: ✓ `bins` array in config specifies which binaries to build: +`cargo-mobile2 = { version = "0.5.2", bins = ["cargo-android", "cargo-mobile"] }`. +Builds all binaries if `bins` omitted. ([cargo-run-bin]) + +**npm**: ✓ `--package` (or `-p`) specifies package when binary name differs: +`npx --package=foo bar`. npx auto-matches binary to package name; errors if no +match. `bin` field in package.json maps command names to scripts. ([npx docs]) + +#### Tool discovery + +How scripts and editors programmatically find paths to installed tool +executables. + +**Dune** (Req 2.2): `dune tools path ` returns the executable path for a +specific tool. Per-tool granularity enables editors to find exact binaries. + +**uv**: ✓ `uv tool dir --bin` returns the bin directory (e.g., `~/.local/bin`). +`uv tool list` shows installed tools. No per-tool path command; construct path +as `$(uv tool dir --bin)/`. ([uv docs]) + +**cargo**: ✗ No discovery command. Binaries in `~/.cargo/bin/` (or +`$CARGO_INSTALL_ROOT/bin`). `cargo install --list` shows installed crates but +not paths. Scripts use `which ` or hardcode `~/.cargo/bin/`. ([cargo +docs]) + +**cargo-run-bin**: ✗ Binaries cached in `.bin/` directory. No CLI path command; +library API (`binary::install()`) returns paths programmatically. +([cargo-run-bin]) + +**npm**: ✓ `npm bin` returns `node_modules/.bin` directory (removed in npm 9+). +npm scripts automatically get `.bin` in PATH. No per-tool path command; use +`$(npm bin)/` or `./node_modules/.bin/`. ([npm docs: bin]) + +#### Tool upgrade + +Update installed tools to newer versions, optionally preserving version +constraints. + +**Dune** (Non-req 6.2): ✗ No upgrade command. Re-run `dune tools add ` to +get latest version, or `dune tools add .` for specific version. +Constraints from `(tool)` stanza preserved on re-lock. + +**uv**: ✓ `uv tool upgrade ` or `--all`. Respects original constraints: if +installed with `black>=23,<24`, upgrade stays within that range. To change +constraints, must reinstall with `uv tool install`. ([uv docs]) + +**cargo**: ✓\* `cargo install ` auto-detects newer versions without +`--force`. No constraint preservation; always installs latest (or specified) +version. Third-party `cargo-update` provides `cargo install-update -a` for batch +upgrades. ([cargo docs]) + +**cargo-run-bin**: ✗ No upgrade command. Edit version in `Cargo.toml` manually, +then run `cargo bin --install`. Version-controlled config ensures team sync. +([cargo-run-bin]) + +**npm**: ✓ `npm update` upgrades within semver constraints (e.g., `^1.2.3` +allows minor/patch updates). `npm outdated` shows available updates. For major +version bumps beyond constraints, use third-party `npm-check-updates`. ([npm +docs: update]) + +#### System-wide install + +Install tools globally, available outside any specific project. + +**Dune**: ✗ Intentionally omitted. Tools are workspace-local only. (See +[Non-requirement 6.1](#61-system-wide-installation)) + +**uv**: ✓ `uv tool install` adds to PATH. "Executables are installed to the uv +tools directory." ([uv docs]) + +**cargo**: ✓ `cargo install` puts binaries in `~/.cargo/bin/`, typically on +PATH. ([cargo docs]) + +**cargo-run-bin**: ✗ Project-local only. + +**npm**: ✓ `npm install -g` installs globally to system. ([npm docs]) + +### Recommendations from Research + +CR-soon Alizter: Review the entire Comparison with Other Tools section for +accuracy and completeness. Verify citations are still valid and that the +comparison table reflects current design decisions. + +**Adopted in this design**: + +- **Batch install** (from cargo-run-bin, npm): `dune tools add` with no + arguments locks all tools from `(tool)` stanzas (Req 4.1) +- **Declarative config** (from npm, cargo-run-bin): `(tool)` stanza in + `dune-workspace` (Req 4.2) +- **Binary selection** (from cargo-run-bin): `(executable ...)` field and + `--bin` flag (Req 1.6) + +**Future consideration**: + +- **`--locked` flag** (from cargo-run-bin): Ensure exact versions from solver, + not just constraints +- **Ephemeral run mode** (from uv's `uvx`): See + [Non-requirement 6.6](#66-ephemeral-runs) +- **System-wide linking** (from uv): See + [Non-requirement 6.1](#61-system-wide-installation) + +[uv docs]: https://docs.astral.sh/uv/concepts/tools/ +[uv docs: uvx]: https://docs.astral.sh/uv/concepts/tools/#running-tools +[cargo docs]: https://doc.rust-lang.org/cargo/commands/cargo-install.html +[cargo#5120]: https://github.com/rust-lang/cargo/issues/5120 +[rustup docs]: https://rust-lang.github.io/rustup/overrides.html +[cargo-run-bin]: https://github.com/dustinblackman/cargo-run-bin +[npm docs]: https://docs.npmjs.com/cli/ +[npm docs: folders]: https://docs.npmjs.com/cli/v9/configuring-npm/folders +[npm docs: package.json]: + https://docs.npmjs.com/cli/v10/configuring-npm/package-json +[npm docs: bin]: https://docs.npmjs.com/cli/v8/commands/npm-bin +[npm docs: update]: https://docs.npmjs.com/cli/v10/commands/npm-update +[npx docs]: https://docs.npmjs.com/cli/v10/commands/npx +[npm v7 changes]: + https://blog.npmjs.org/post/626173315965468672/npm-v7-series-beta-release-and-semver-major.html + +## Learnings from the Prototype + +CR-soon Alizter: This entire section may be considered garbage. Extract +important information out into the spec or remove entirely. + +This section documents key insights from the initial prototype implementation. + +### What Worked Well + +1. **Versioned lock directories**: Storing tools in + `_build/.tools.lock///` cleanly separates versions and keeps + the source tree clean. The flow (solve → extract version → move to versioned + path) is straightforward. + +2. **Tool isolation via synthetic wrapper**: Solving tools with a synthetic + `_tool_wrapper` package (not mixed with project packages) prevents + "packages outside workspace" solver errors. + +3. **Install cookie as build target**: Depending on `target/cookie` rather than + individual binaries correctly handles directory targets. The cookie is + created after full installation, providing a reliable completion signal. + +4. **Disk reads for lock checks**: Reading lock directories directly from disk + (not via build system) avoids triggering expensive project builds during tool + commands. + +5. **Simple PATH-only environment**: Tools only need the bin directory in PATH, + not a full exported environment. This avoids expensive closure computations. + +6. **Unified resolution with fallback**: A single resolution chain (stanza → + locked → PATH) provides clear priority and graceful degradation. + +### Problems Discovered + +1. **Directory targets cannot be depended upon directly**: Naively depending on + `_build/.../target/bin/` fails. Solution: depend on cookie, then read + cookie to discover binaries. + +2. **Build system triggers on lock checks**: Calling `Lock_dir.get` triggered + full builds. Solution: use `Lock_dir.read_disk` for checking. + +CR-soon Alizter: Need to understand what this is about. + +3. **Checksum collection missed versioned paths**: Fetch rules initially only + collected unversioned paths. Solution: scan `.tools.lock/` for all versions. + +4. **`.pkg/` rules gated by project lock**: Initial design skipped rule + generation when no project lock existed, breaking tools-only projects. + Solution: always generate rules; create tools-only DB when needed. + +5. **Compiler matching requires explicit package**: Just constraining `ocaml` + version wasn't enough; solver picked `ocaml-base-compiler` instead of + `ocaml-system`. Solution: explicitly require `ocaml-system` when system + compiler is available. + +### Reference Packages for Testing + +The implementation should be tested against these packages. We don't +special-case them, but they represent common tool patterns and historic +dev-tools support. + +CR-soon Alizter: Vet this list. Are there other common tools to add? Any that +should be removed? Consider tools from opam-repository's most-installed +packages. + +**Historic dev-tools** (from `src/dune_pkg/dev_tool.ml`): + +| Package | Binary | Notes | +| ------------------ | ---------------- | --------------------------------- | +| `ocamlformat` | `ocamlformat` | Also has `ocamlformat-rpc` binary | +| `odoc` | `odoc` | Documentation generator | +| `ocaml-lsp-server` | `ocamllsp` | Binary name differs from package | +| `utop` | `utop` | REPL, needs project libraries | +| `earlybird` | `ocamlearlybird` | Debugger | +| `odig` | `odig` | Documentation viewer | +| `opam-publish` | `opam-publish` | Release tooling | +| `dune-release` | `dune-release` | Release tooling | +| `ocaml-index` | `ocaml-index` | Indexer for ocamllsp | +| `merlin` | `ocamlmerlin` | Editor support | + +CR-soon Alizter: These are not historic dev tools, these are dev tools that we +currently support and would like to continue to support. Linking this to the +migration requirement. They will serve as test cases sure, but they also serve +as platform roadmap integration etc. + +**Edge cases to verify**: + +- Package with single binary: auto-selection works +- Package with multiple binaries: error with helpful message +- Package name ≠ binary name: `--bin` or `(executable)` works +- Tool needing project context: `utop` with `#require` + +CR-soon Alizter: I don't like these verifications here, they are out of place + +## Ideas + +CR-someday Alizter: Consider supporting `(tool)` stanzas in `dune-project` files +for per-project tool configuration, similar to how `(pin)` exists in both +`dune-workspace` and `dune-project`. This would allow different projects within +a workspace to declare different tool requirements. Conflict resolution: +workspace wins over project, outer project wins over nested. Sibling projects +with conflicting declarations can coexist. Within project-scoped rules the +correct tool is unambiguous. CLI commands without project context (e.g., +`dune tools run` from workspace root) should error if declarations conflict. + +CR-someday Alizter: Multi-version scaling. Supporting N versions means N +separate solves, downloads, and builds. In a monorepo migrating ocamlformat +versions across 50 sub-projects, users could accumulate many versions. Consider: +(1) warning when version count exceeds threshold, (2) `dune tools gc` to remove +unused versions, (3) documenting expected steady-state (few versions, not many). + +### Editor integration + +Editors and IDEs must be able to: + +- Discover which tools are available in a workspace +- Find the path to tool executables for spawning +- Invoke tools with dune managing the process (alternative pattern) + +See [CLI commands](./implementation.md#cli-commands) for the discovery and invocation interfaces. + +**Open issues** (require consultation with editors team): + +- How editors discover which tools are available +- LSP server invocation pattern: path discovery (editor spawns process) vs + managed invocation (dune manages process). LSP servers need editor control of + stdin/stdout for JSON-RPC and process lifecycle (restart on crash). +- Single-file formatting: `dune fmt ` for editor integration ([#3244] + discusses this) + +[#3244]: https://github.com/ocaml/dune/issues/3244 + +- `dune fmt` over RPC for watch mode integration +- Integration with `.ocamlformat` version detection +- Recommended editor configuration patterns + +CR-someday Alizter: Path staleness problem. If an editor caches the result of +`dune tools path ocamllsp` and the user later updates their OCaml version, the +cached path may point to an ABI-incompatible binary. There's no mechanism to +notify editors that cached paths are invalid. Options: (1) editors always call +`dune tools path` fresh, (2) dune provides a staleness check, (3) watch mode +integration notifies editors of tool rebuilds. + From 8d3c225a98fbb4d225426cffb7f18aee83cd9dcc Mon Sep 17 00:00:00 2001 From: Sudha Parimala Date: Tue, 28 Jul 2026 11:29:14 +0200 Subject: [PATCH 70/72] Update TOC generation script Signed-off-by: Sudha Parimala --- doc/dev/tools-design/implementation.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md index 9258a90fbda..b5525a4ccc2 100644 --- a/doc/dev/tools-design/implementation.md +++ b/doc/dev/tools-design/implementation.md @@ -7,7 +7,7 @@ Authors: Ali Caglayan (Tarides), Shon Feder (Tarides) This document specifies the implementatino of dune's tool management system. @@ -15,6 +15,7 @@ nix shell --impure --expr 'let pkgs = import (builtins.getFlake "github:NixOS/ni - [Tools Implementation](#tools-implementation) - [Summary](#summary) +- [How to Read This Document](#how-to-read-this-document) - [Design](#design) - [The `(tool)` stanza](#the-tool-stanza) - [Syntax](#syntax) From 9a034f2607b12efeff76215745778f2745c6bc40 Mon Sep 17 00:00:00 2001 From: Sudha Parimala Date: Tue, 28 Jul 2026 12:32:28 +0200 Subject: [PATCH 71/72] Extend tool stanza to include more details This is a rewrite of the tool stanza with the following: - Include relevant fields from lock_dir stanza. - Introduce a special variable :inhert, and a field (inherit_lock_dir) to inherit config from lock_dir stanza. - More examplesWq Signed-off-by: Sudha Parimala --- doc/dev/tools-design/implementation.md | 134 +++++++++++++++++++++---- 1 file changed, 114 insertions(+), 20 deletions(-) diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md index b5525a4ccc2..84932a4e9f0 100644 --- a/doc/dev/tools-design/implementation.md +++ b/doc/dev/tools-design/implementation.md @@ -127,16 +127,22 @@ uses the same dependency syntax as `(depends ...)` for the package field. ```lisp (tool - (package ) ; Required: package name with optional constraint - (executable ) ; Optional: binary name, defaults to package name - (skip_compiler_match) ; Optional: flag to disable compiler matching - (repositories ...)) ; Optional: restrict to specific repositories + (package ) ; required: the tool package + (executables ...) ; optional: binaries to expose + (inherit_lock_dir []) ; optional: inherit solve env from a lock_dir stanza + (repositories |:inherit ...) ; optional: repositories to solve from + (solver_env ) ; optional: solver variable assignments + (unset_solver_vars |:inherit ...) ; optional: solver variables to leave undefined + (version_preference ) ; optional + (solve_for_platforms |:inherit ...) ; optional: platforms to solve for + (constraints |:inherit ...) ; optional: constraints on each solve + (pins |:inherit ...) ; optional: pins (names of (pin ...) stanzas) + (skip_compiler_match)) ; optional: disable compiler matching ``` -CR-soon Alizter: missing pin +CR-soon Sudha247: Can we express skip_compiler_match within the constraints field somehow? -CR-soon Alizter: Should share some of lock_dir stanza fields. Need to do a -comparison and explain why it is or is not a good idea. +CR-soon Sudha247: do we need a `depopts` field? CR-soon Alizter: Edge cases for stanza parsing: @@ -149,6 +155,21 @@ CR-soon Alizter: Edge cases for stanza parsing: CR-soon Sudha247: Capture constraints here? Related issue: [dune#12777](https://github.com/ocaml/dune/issues/12777) +#### Semantics of `:inherit` + + - Valid only in list-valued fields (repositories, unset_solver_vars, + solve_for_platforms, constraints, pins), and expands in place to the + corresponding field's value from the stanza named by `inherit_lock_dir`, e.g. + (repositories :inherit my-repo) means the inherited repositories followed by + my-repo. + - Omitting a field entirely means fully inherit it (when inherit_lock_dir is + present); writing the field without :inherit means fully override it. + - Using :inherit without inherit_lock_dir is an error. + - solver_env and version_preference are scalar-ish, so they follow the + omit-to-inherit / write-to-override rule with no :inherit keyword. + - Having an empty field with an existing (inherit_lock_dir) means omit that + particular field + #### Fields - **`(package )`** (required): The opam package providing the tool. Accepts @@ -158,23 +179,70 @@ CR-soon Sudha247: Capture constraints here? Related issue: - Version operators: `=`, `<>`, `<`, `>`, `<=`, `>=` - Conjunctions: `(and (>= 0.25.0) (< 0.27.0))` -- **`(executable )`** (optional): The binary to run. Defaults to the - package name. Required when the package provides multiple binaries. +- **`(executables )`** (optional): The binary to run. Defaults to the + package name. If the package installs multiple binaries and no binary matches + the package name, omitting this field is an error. + +- **`(inherit_lock_dir [])`** (optional): Inherit the solve + environment from the `(lock_dir)` stanza whose path is `` + (defaulting to `dune.lock`, the default lock directory). The inheritable + fields are `repositories`, `solver_env`, `unset_solver_vars`, + `version_preference`, `solve_for_platforms`, `constraints`, and `pins`. + + When this field is present, each inheritable field follows one rule: + + | You write | Meaning | + |--------------------------|----------------------------------------------| + | *(field omitted)* | fully inherit the lock_dir's value | + | `(field )` | fully override; inheritance suppressed | + | `(field :inherit ...)` | splice the lock_dir's value in at that position | + | `(field :standard ...)` | splice dune's built-in default in | + + `:inherit` is only valid in list-valued fields, and only when + `inherit_lock_dir` is present; using it without is an error. Without + `inherit_lock_dir`, omitted fields take Dune's ordinary defaults. + +- **`(repositories |:inherit|:standard ...)`** (optional): + Repositories to solve from, as an ordered set. `:standard` is the default + repository set; `:inherit` is the lock_dir's repositories. Defaults to + `:inherit` when `inherit_lock_dir` is present, `:standard` otherwise. + +- **`(solver_env )`** (optional): Solver variable assignments for + this tool's solve. Scalar semantics: omitted = inherited (when + `inherit_lock_dir` is present), written = taken exactly as written, + `(solver_env)` = empty. To inherit the environment *minus* a variable, + inherit it and list the variable in `unset_solver_vars`. + +- **`(version_preference )`** (optional): Whether the solver + prefers the newest or oldest satisfying versions. Omitted = inherited + (when `inherit_lock_dir` is present), otherwise `newest`. To suppress + inheritance, spell the default: `(version_preference newest)`. + +- **`(solve_for_platforms |:inherit|:standard ...)`** (optional): + Platforms to solve this tool for. `:standard` is the default platform set; + `:inherit` is the lock_dir's platforms. Writing the field so that it + evaluates to an empty set is an error. To get the default platforms + explicitly, write `(solve_for_platforms :standard)`. + +- **`(constraints |:inherit ...)`** (optional): Additional constraints + on the solve, in `(depends ...)` syntax. A constraint only binds when the + named package actually appears in the tool's solution, so inheriting the + project's constraints is safe: irrelevant ones are no-ops, and shared + transitive dependencies stay consistent with the project. + +- **`(pins |:inherit ...)`** (optional): Names of `(pin ...)` + stanzas to apply to this tool's solve. - **`(skip_compiler_match)`** (optional): When present, disables the default compiler matching behavior. Use this for tools that don't need compiler compatibility (e.g., formatters that only parse source text). -CR-soon Alizter: The prototype uses `compiler_compatible` (opt-in) but this doc -says `skip_compiler_match` (opt-out). Reconcile naming and semantics. - -- **`(repositories )`** (optional): Restrict package resolution to the - named repositories. Useful for binary package repositories. - CR-soon Alizter: The `(repositories)` field is parsed but verify that the restriction is actually applied during solving in the prototype. -CR-soon Alizter: Missing pins +CR-soon Alizter: The prototype uses `compiler_compatible` (opt-in) but this doc +says `skip_compiler_match` (opt-out). Reconcile naming and semantics. + #### Compiler matching @@ -223,17 +291,43 @@ CR-soon Alizter: Edge cases for compiler matching: ;; Specifying the binary for multi-binary packages (tool (package menhir) - (executable menhirSdk)) + (executables menhirSdk)) ;; Tool that doesn't need compiler matching (tool (package ocamlformat) (skip_compiler_match)) -;; Using a binary package repository +;; Tool pinned to a fork +(pin + (name ocamlformat) + (url "git+https://github.com/me/ocamlformat#my-fork")) + +(tool + (package ocamlformat) + (pins ocamlformat)) + +;; Inherit from lock_dir (tool - (package ocaml-lsp-server) - (repositories binary-packages)) + (package ocamlformat) + (inherit_lock_dir)) + +;; More involved menhir + +(tool + (package (menhir (>= 20220210))) + (executables menhir menhirSdk) + (inherit_lock_dir dev.lock) + (repositories :inherit overlay-repo) ; dev.lock's repos, then overlay-repo + (constraints :inherit (dune-build-info (>= 3.0))) ; project constraints plus one + (solver_env + (with-test false)) ; full override: dev.lock's solver_env + ignored + (version_preference oldest) + (solve_for_platforms :standard) ; opt out of dev.lock's platforms, use dune's + defaults + (skip_compiler_match)) + ``` #### Invalid tool selections From 8ba90ebb4725f5a1294d9052190f0ddb3b1096b9 Mon Sep 17 00:00:00 2001 From: Sudha Parimala Date: Wed, 29 Jul 2026 10:56:58 +0200 Subject: [PATCH 72/72] Add CR about dune tools commands editing the dune-workspace Signed-off-by: Sudha Parimala --- doc/dev/tools-design/implementation.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/dev/tools-design/implementation.md b/doc/dev/tools-design/implementation.md index 84932a4e9f0..7d293aab6b4 100644 --- a/doc/dev/tools-design/implementation.md +++ b/doc/dev/tools-design/implementation.md @@ -410,12 +410,20 @@ CR-soon Alizter: `dune tools path` behavior is underspecified: - How does `path` interact with `(tool)` stanzas vs CLI-added tools? Same resolution as `run`? +CR-soon Sudha247: We now have two sources for the current state of what tools +exist - one in the declarative config within dune-workspace, and one in the +state of tools installed and their lockdirs. Ideally we should just have a +single source of truth for what tools should be installed. We can do this by +making the CLI edit dune-workspace rather than operating on its own. A +pre-requisite for this is: https://github.com/ocaml/dune/pull/13758. + CR-someday Alizter: Consider a single bin directory with symlinks to all tool executables (like npm's `node_modules/.bin/`). Instead of per-tool paths, have `_build/.tools/bin/` containing symlinks to all installed tool binaries. This simplifies editor integration (one directory to add to PATH), avoids PATH length limits (especially on Windows), and provides a single stable location for discovery. Trade-off: need to maintain symlinks as tools are added/removed. + #### Version syntax The CLI uses dot-separated version syntax: `.`