refactor: reimplement writing namespace pkgs in Starlark #2882

aignas · 2025-05-15T14:05:25Z

With this PR I would like to facilitate the implementation of the venv
layouts because we can in theory take the srcs and the data within
the py_library and then use the expand_template to write the extra
Python files if the namespace_pkgs flag is enabled.

The old Python code has been removed and the extra generated files are
written out with bazel_skylib copy_file.

The implicit namespace_pkg init files are included to py_library
if the site-packages config flag is set to false and I think this
may help with continuing the implementation, but it currently is still
not working as expected (see comment).

Work towards #2156

rickeylev · 2025-05-16T19:42:44Z

This is basically a reimplementation of https://github.com/bazel-contrib/rules_python/blob/main/python/private/pypi/whl_installer/namespace_pkgs.py right?

Mostly LGTM.

Is it handling the "# The root of the directory should never be an implicit namespace" condition?

remove the python code, put under pipstar

I think you mean: "if pipstar is enabled, don't run the python version"

This doesn't have to be tied to pipstar. It can just be moved to be part of the pip-generated BUILD files. e.g.

py_library(
  name = "pkg",
  srcs = [":_pkgutil_inits"] + glob(**.py)
)

pkgutil_inits( # produces 0 or more pkgutil init.py files based on what it sees in srcs
  name = "_pkgutil_inits",
  srcs = glob("**")
)

aignas · 2025-05-18T03:14:29Z

Ran:

$ bazel build //docs --@rules_python//python/config_settings:venvs_site_packages=yes
INFO: Analyzed target //docs:docs (0 packages loaded, 0 targets configured).
ERROR: /home/aignas/src/github/aignas/rules_python/docs/BUILD.bazel:45:12: Sphinx building html for //docs:docs failed: (Exit 2): sphinx-build failed: error executing SphinxBuildDocs command (from target //docs:docs) bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build --show-traceback --builder html --quiet --jobs auto --fresh-env --write-all bazel-out/k8-fastbuild/bin/docs/_docs/_sources ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging

Traceback (most recent call last):
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/sphinx/cmd/build.py", line 496, in build_main
    app = Sphinx(
          ^^^^^^^
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/sphinx/application.py", line 256, in __init__
    self.setup_extension(extension)
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/sphinx/application.py", line 437, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/sphinx/registry.py", line 466, in load_extension
    metadata = setup(app)
               ^^^^^^^^^^
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/__init__.py", line 8, in setup
    from .sphinx.extension import setup as _setup
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/sphinx/extension.py", line 18, in <module>
    from autodoc2.sphinx.autodoc import AutodocObject
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/sphinx/autodoc.py", line 12, in <module>
    from autodoc2.sphinx.utils import (
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/sphinx/utils.py", line 15, in <module>
    from autodoc2.utils import WarningSubtypes
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/utils.py", line 10, in <module>
    from typing_extensions import Required
ModuleNotFoundError: No module named 'typing_extensions'

Exception occurred:
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/utils.py", line 10, in <module>
    from typing_extensions import Required
ModuleNotFoundError: No module named 'typing_extensions'
The full traceback has been saved in /tmp/sphinx-err-twbss_8e.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Target //docs:docs failed to build
Use --verbose_failures to see the command lines of failed build steps.

I checked the venv layout:

$ ll bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/_sphinx-build.venv/lib/python3.11/site-packages
.r-xr-xr-x 8.4k aignas 18 May 12:07 _bazel_site_init.py
lrwxrwxrwx    - aignas 18 May 12:00 _yaml -> ../../../../../../+pip+dev_pip_311_pyyaml_cp311_cp311_manylinux_2_17_x86_64_3ad2a3de/site-packages/_yaml
lrwxrwxrwx    - aignas 18 May 12:00 alabaster -> ../../../../../../+pip+dev_pip_311_alabaster_py3_none_any_fc678640/site-packages/alabaster
lrwxrwxrwx    - aignas 18 May 12:00 astroid -> ../../../../../../+pip+dev_pip_311_astroid_py3_none_any_d05bfd0a/site-packages/astroid
lrwxrwxrwx    - aignas 18 May 12:00 autodoc2 -> ../../../../../../+pip+dev_pip_311_sphinx_autodoc2_py3_none_any_e867013b/site-packages/autodoc2
lrwxrwxrwx    - aignas 18 May 12:00 babel -> ../../../../../../+pip+dev_pip_311_babel_py3_none_any_4d0b5309/site-packages/babel
.r-xr-xr-x   24 aignas 12 May 19:45 bazel.pth
lrwxrwxrwx    - aignas 18 May 12:00 certifi -> ../../../../../../+pip+dev_pip_311_certifi_py3_none_any_ca78db45/site-packages/certifi
lrwxrwxrwx    - aignas 18 May 12:00 charset_normalizer -> ../../../../../../+pip+dev_pip_311_charset_normalizer_cp311_cp311_manylinux_2_17_x86_64_fd4ec41f/site-packages/charset_normalizer
lrwxrwxrwx    - aignas 18 May 12:00 docutils -> ../../../../../../+pip+dev_pip_311_docutils_py3_none_any_dafca5b9/site-packages/docutils
lrwxrwxrwx    - aignas 18 May 12:00 idna -> ../../../../../../+pip+dev_pip_311_idna_py3_none_any_946d195a/site-packages/idna
lrwxrwxrwx    - aignas 18 May 12:00 imagesize -> ../../../../../../+pip+dev_pip_311_imagesize_py2_none_any_0d8d18d0/site-packages/imagesize
lrwxrwxrwx    - aignas 18 May 12:00 jinja2 -> ../../../../../../+pip+dev_pip_311_jinja2_py3_none_any_85ece445/site-packages/jinja2
lrwxrwxrwx    - aignas 18 May 12:00 markdown_it -> ../../../../../../+pip+dev_pip_311_markdown_it_py_py3_none_any_35521684/site-packages/markdown_it
lrwxrwxrwx    - aignas 18 May 12:00 markupsafe -> ../../../../../../+pip+dev_pip_311_markupsafe_cp311_cp311_manylinux_2_17_x86_64_a123e330/site-packages/markupsafe
lrwxrwxrwx    - aignas 18 May 12:00 mdit_py_plugins -> ../../../../../../+pip+dev_pip_311_mdit_py_plugins_py3_none_any_0c673c3f/site-packages/mdit_py_plugins
lrwxrwxrwx    - aignas 18 May 12:00 mdurl -> ../../../../../../+pip+dev_pip_311_mdurl_py3_none_any_84008a41/site-packages/mdurl
lrwxrwxrwx    - aignas 18 May 12:00 myst_parser -> ../../../../../../+pip+dev_pip_311_myst_parser_py3_none_any_b9317997/site-packages/myst_parser
lrwxrwxrwx    - aignas 18 May 12:00 packaging -> ../../../../../../+pip+dev_pip_311_packaging_py3_none_any_29572ef2/site-packages/packaging
lrwxrwxrwx    - aignas 18 May 12:00 pygments -> ../../../../../../+pip+dev_pip_311_pygments_py3_none_any_9ea1544a/site-packages/pygments
lrwxrwxrwx    - aignas 18 May 12:00 readthedocs_ext -> ../../../../../../+pip+dev_pip_311_readthedocs_sphinx_ext_py2_none_any_f8c56184/site-packages/readthedocs_ext
lrwxrwxrwx    - aignas 18 May 12:00 requests -> ../../../../../../+pip+dev_pip_311_requests_py3_none_any_70761cfe/site-packages/requests
lrwxrwxrwx    - aignas 18 May 12:00 snowballstemmer -> ../../../../../../+pip+dev_pip_311_snowballstemmer_py2_none_any_c8e1716e/site-packages/snowballstemmer
lrwxrwxrwx    - aignas 18 May 12:00 sphinx -> ../../../../../../+pip+dev_pip_311_sphinx_py3_none_any_09719015/site-packages/sphinx
lrwxrwxrwx    - aignas 18 May 12:00 sphinx_reredirects -> ../../../../../../+pip+dev_pip_311_sphinx_reredirects_py3_none_any_efd50c76/site-packages/sphinx_reredirects
lrwxrwxrwx    - aignas 18 May 12:00 sphinx_rtd_theme -> ../../../../../../+pip+dev_pip_311_sphinx_rtd_theme_py2_none_any_422ccc75/site-packages/sphinx_rtd_theme
drwxr-xr-x    - aignas 18 May 12:00 sphinxcontrib
lrwxrwxrwx    - aignas 18 May 12:00 urllib3 -> ../../../../../../+pip+dev_pip_311_urllib3_py3_none_any_4e166650/site-packages/urllib3
lrwxrwxrwx    - aignas 18 May 12:00 yaml -> ../../../../../../+pip+dev_pip_311_pyyaml_cp311_cp311_manylinux_2_17_x86_64_3ad2a3de/site-packages/yaml

And I think the problem is that the typing-extensions package is just a single file inside site-packages, so it does not work because it is not included in the venv. The build still works if I run it without the flag.

As found in bazel-contrib#2882 testing, packages like `typing-extensions` which have `.py` files at the root of the `site-packages` folder don't work and it seems that the comment about `rules_python` being too eager is only half-correct. Since `namespace_pkgs` are no longer there, we can just include all of the files and if there are collisions, they will be highlighted as build errors. Now the following works: ``` bazel build //docs --@rules_python//python/config_settings:venvs_site_packages=yes ``` Stacked on bazel-contrib#2882. Work towards bazel-contrib#2156

With this PR I would like to facilitate the implementation of the venv layouts because we can in theory take the srcs and the data within the py_library and then use the expand_template to write the extra Python files if the namespace_pkgs flag is enabled. The old Python code has been removed and the extra generated files are written out with bazel_skylib write_file as a POC. This could be optimized by writing once and then doing symlinks to the original instead of writing the files, but I am not sure how this would behave, maybe fine? The implicit namespace_pkg init files are included to py_library if the site-packages config flag is set to false and I think this may help with continuing the implementation. NOTE, it seems that the pycross code that we have is using the namespace_pkg Python code, so that will be removed later in one PR. Work towards #2156

As found in bazel-contrib#2882 testing, packages like `typing-extensions` which have `.py` files at the root of the `site-packages` folder don't work and it seems that the comment about `rules_python` being too eager is only half-correct. Since `namespace_pkgs` are no longer there, we can just include all of the files and if there are collisions, they will be highlighted as build errors. Now the following works: ``` bazel build //docs --@rules_python//python/config_settings:venvs_site_packages=yes ``` Stacked on bazel-contrib#2882. Work towards bazel-contrib#2156

aignas requested review from rickeylev and groodt as code owners May 15, 2025 14:05

aignas mentioned this pull request May 15, 2025

Implement venv/site-packages based binaries #2156

Open

10 tasks

aignas changed the title ~~wip: reimplement namespace pkg filepath calc in starlark~~ refactor: reimplement writing namespace pkgs in Starlark May 17, 2025

aignas force-pushed the refactor/namespace-pkg-star branch 2 times, most recently from 54136c5 to 98827e5 Compare May 18, 2025 03:01

aignas force-pushed the refactor/namespace-pkg-star branch from 98827e5 to 4f1798b Compare May 18, 2025 03:20

aignas mentioned this pull request May 18, 2025

fix: symlink root-level python files to the venv #2908

Open

aignas force-pushed the refactor/namespace-pkg-star branch from bb150a7 to 4ed19e0 Compare May 18, 2025 06:27

aignas enabled auto-merge May 18, 2025 06:28

Merge branch 'main' into refactor/namespace-pkg-star

5a28b4b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: reimplement writing namespace pkgs in Starlark #2882

refactor: reimplement writing namespace pkgs in Starlark #2882

aignas commented May 15, 2025 •

edited

Loading

rickeylev commented May 16, 2025

aignas commented May 18, 2025

refactor: reimplement writing namespace pkgs in Starlark #2882

Are you sure you want to change the base?

refactor: reimplement writing namespace pkgs in Starlark #2882

Conversation

aignas commented May 15, 2025 • edited Loading

rickeylev commented May 16, 2025

aignas commented May 18, 2025

aignas commented May 15, 2025 •

edited

Loading