pip install -e ".[dev]"

[petrasvestartas]

Hi,

I want to use the following command to install dependencies from both requirements.txt and requirements-dev.txt:

pip install -e ".[dev]"

I understand that this can be achieved by specifying the dependencies in pyproject.toml. However, I haven’t found a way to install them directly from the text files when using scikit-build-core.

Do you have any recommendations or insights into why pip install -e ".[dev]" doesn't work with scikit-build-core unless the dependencies are explicitly listed in pyproject.toml, rather than being pulled from the requirements text files?

dependencies = [
    "numpy >= 1.21.0",
    "compas >= 2",
    "compas_dem",
]


[project.optional-dependencies]
dev = [
    "attrs >= 17.4",
    "black >= 22.12.0",
    "bump-my-version",
    "compas_invocations2",
    "invoke >= 0.14",
    "ruff",
    "sphinx_compas2_theme",
    "twine",
    "wheel",
    "scikit-build-core",
    "nanobind",
    "cmake",
    "build",
    "compas_viewer",
]

[LecrisUT]

Any reason why you want to have a requirements.txt file? It is pretty much deprecated nowadays especially with the support of dependency groups. In order to support requirements.txt we need to explicitly add that support in and on your side make all depdendencies and optional dependencies dynamic.

Instead there are tools out there that can generate requirements.txt lock files like pip-compile, or maybe even more generic ones

[henryiii]

You can use dynamic-metadata to implement this. Currently it still requires the experimental=True toggle. It will require fully dynamic optional-dependencies too; I have a PEP that would fix this, but waiting on @FFY00 to review. :)

For clarification of @LecrisUT's point, this is what it should look like with modern packaging:

dependencies = [
    "numpy >= 1.21.0",
    "compas >= 2",
    "compas_dem",
]


[dependency-groups]
dev = [
    "attrs >= 17.4",
    "black >= 22.12.0",
    "bump-my-version",
    "compas_invocations2",
    "invoke >= 0.14",
    "ruff",
    "sphinx_compas2_theme",
    "twine",
    "wheel",
    "scikit-build-core",
    "nanobind",
    "cmake",
    "build",
    "compas_viewer",
]

pip install -e. --group dev

(This will even automatically work with the high-level uv interface, like uv run and uv sync).

[petrasvestartas]

Thank you

Where should I write experimental=True?

[henryiii]

# pyproject.toml
[tool.scikit-build]
experimental = true

[tool.scikit-build.metadata.optional-dependencies]
provider = "devdeps"
provider-path = "tools"

# tools/devdeps.py
from collections.abc import Mapping
from typing import Any
from pathlib import Path

def dynamic_metadata(
    field: str,
    settings: Mapping[str, Any],
    project: Mapping[str, Any],
) -> str:
    assert field == "optional-dependencies"

    requirements = [r for s in Path("requirements.txt").read_text().splitlines() if (r := s.strip().split("#")[0])]
    return {"dev": requirements}

Something like that.

[petrasvestartas]

Thank you very much!

[petrasvestartas]

Is it fair to say that when we do these kind of wrappers, we have 1) local build system 2) for distribution we have cibuildwheel system?

And how they are configure on the pyproject.toml must also treated separately?

Github actions currently also set like that:
https://github.com/compas-dev/compas_cgal/actions/runs/15899882696

[LecrisUT]

Is it fair to say that when we do these kind of wrappers, we have 1) local build system 2) for distribution we have cibuildwheel system?

And how they are configure on the pyproject.toml must also treated separately?

Github actions currently also set like that: https://github.com/compas-dev/compas_cgal/actions/runs/15899882696

I'm afraid I don't quite follow. Could you break it down more piecewise what you are trying to ask/do?

[petrasvestartas]

We use this workflow for local non c++ related projects:

# ============================================================================
# setuptools config
# ============================================================================

[tool.setuptools.dynamic]
dependencies = { file = "requirements.txt" }
optional-dependencies = { dev = { file = "requirements-dev.txt" } }

For cibuildwheel and tests we need again to specify these requirements, look at the list of test-requires:

[tool.cibuildwheel]
build-verbosity = 3
test-requires = ["numpy", "compas", "pytest", "build", "tessagon"]
test-command = "pip install numpy compas && pip list && pytest {project}/tests"
build-frontend = "pip"
manylinux-x86_64-image = "manylinux2014"
skip = ["*_i686", "*-musllinux_*", "*-win32", "pp*"]
macos.environment.MACOSX_DEPLOYMENT_TARGET = "11.00"
macos.archs = ["x86_64", "arm64"]

I am wondering why this cannot be one and the same thing.

Full project:

https://github.com/compas-dev/compas_libigl/blob/main/pyproject.toml

[LecrisUT]

For cibuildwheel you can change the test-requires to be test-extras = "dev"

But, please be kind to the downstream packagers and split up the dev into more logical components like test, lint etc.

Stuff like nanobind, scikit-build-core are unnecessary to be added in the extras because they are already resolved from the build-system.requires. Stuff like twine and build are better had in dependency-groups because that one will not build your package as part of the installation, unlike the extras, and you need those tools before you even do the build.

[henryiii]

The recommendation would be as follows:

Don't use requirements.txt: setuptools strongly discourages using the setting above for this sort of thing; it was added for very specialized cases. Normally, requirements.txt should be a locked output from requirements.in, and not related to packaging at all (and tools like pdm, uv, etc all have better ways to do this these days)

Use project.dependencies for things your package needs in order to run. Usually things like numpy.

Use project.optional-dependencies for things a user might need, but not always. This is often something like [cli] to add dependencies for an optional CLI interface, for example. If you have integrated tests (part of your wheel) that have extra dependencies, that's where you could use [test]. I would never provide a [dev].

Use dependency-groups for things you want to install as part of your package's workflow, but are not "external", that is, users pip installing your package from PyPI wouldn't need them. This is often tests (when your tests live in the SDist but not the wheels), docs, dev, and similar.

Note

I don't add tools to dev, like ruff and pre-commit, since I prefer to install them as tools, with uv tool install or uvx (or via brew), and it's silly to install them every time you use uv run, IMO, but it's not terrible and probably the best we have currently. Sometimes these do conflict! black and tensorflow could not be installed into the same environment at one point. I included them below, though, to match what you were doing.

Here's a hypothetical example, with a little help from ChatGPT (which didn't understand the dependency-groups part because it was too new):

[project]
dependencies = [
    "numpy >=2",
]

[project.optional-dependencies]
cli = [
    "click >=8",
]

[dependency-groups]
dev = [
    "ruff",
    "pre-commit",
    "build",
    { include-group = "tests" },
    { include-group = "docs" },
]
tests = [
    "pytest>=8",
    "pytest-mock",
    "coverage[toml]",
]
docs = [
    "sphinx",
    "furo",
    "sphinx-autodoc-typehints",
]

[tool.cibuildwheel]
test-groups = ["test"]

Now, you can (uv) pip install -e. --group dev (or any other group). You don't have to install the package, say for docs, like (uv) pip install --group docs. In cibuildwheel, you can set test-groups. Since you can include one group in others, this can be far more granular than the other methods, including just the right thing in the right places, which saves time and reduces conflict opportunities.