py_binary: Broken venv symlinks with venvs_site_packages=yes #3388
Description
🐞 bug report
Affected Rule
py_binary when used with py_image_layer rule from aspect_rules_py.
Is this a regression?
I don't know.
I'm not sure if this is a rules_python issue or aspect_rules_py issue, but I know that what I'm doing works with @rules_python//python/config_settings:venvs_site_packages=no and fails with @rules_python//python/config_settings:venvs_site_packages=yes, which is why I'm reporting it here.
Description
When using aspect_rules_py and rules_oci to package a py_binary into a container image, the build fails when @rules_python//python/config_settings:venvs_site_packages=yes, even though it works with @rules_python//python/config_settings:venvs_site_packages=no. More specifically, I'm using py_image_layer rule from aspect_rules_py to package the python binary, and it fails to create the tarball. The error suggests that a file inside the virtual environment created by py_binary cannot be found. The symlink clearly exists yet its target does not (see the errors below).
🔬 Minimal Reproduction
The reproduction is in the examples/oci-container directory of commit 2c596f17cf08 (main...hartikainen:rules_python:oci-container-example).
To reproduce, run bazel build //:server_layer from the example folder, with the different flag combinations in .bazelrc as described below.
🔥 Exception or Error
Setting 1 (success ✅)
# .bazelrc
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=no
common --@rules_python//python/config_settings:venvs_use_declare_symlink=noDetails
$ cd ./examples/oci-container
$ bazel build //:server_layer
WARNING: Build options --@@rules_python+//python/config_settings:venvs_site_packages and --@@rules_python+//python/config_settings:venvs_use_declare_symlink have changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //:server_layer (2 packages loaded, 6998 targets configured).
INFO: Found 1 target...
Target //:server_layer up-to-date:
bazel-bin/server_layer_interpreter.tar.gz
bazel-bin/server_layer_packages.tar.gz
bazel-bin/server_layer_default.tar.gz
INFO: Elapsed time: 24.130s, Critical Path: 23.71s
INFO: 41 processes: 5 action cache hit, 10 internal, 31 processwrapper-sandbox.
INFO: Build completed successfully, 41 total actionsSetting 2 (failure ❌)
# .bazelrc
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=noDetails
$ cd ./examples/oci-container
$ bazel build //:server_layer
INFO: Analyzed target //:server_layer (146 packages loaded, 6968 targets configured).
ERROR: /home/user/bazel-contrib/rules_python/main/examples/oci-container/BUILD.bazel:20:15: Tar server_layer_packages.tar.gz failed: (Exit 1): tar failed: error executing Tar command (from target //:server_layer_packages) external/aspect_bazel_lib++toolchains+bsd_tar_linux_amd64/tar --create --gzip '--options=gzip:!timestamp' --file bazel-out/k8-fastbuild/bin/server_layer_packages.tar.gz ... (remaining 1 argument skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
tar: Error reading archive bazel-out/k8-fastbuild/bin/server_layer.packages.manifest.spec: Can't open bazel-out/k8-fastbuild/bin/_server.venv/lib/python3.12/site-packages/absl
tar: Error exit delayed from previous errors.
Target //:server_layer failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 25.961s, Critical Path: 23.29s
INFO: 62 processes: 56 internal, 6 processwrapper-sandbox.
ERROR: Build did NOT complete successfully
$ readlink -f ./bazel-out/k8-fastbuild/bin/_server.venv/lib/python3.12/site-packages/absl # empty
$ ls -lah ./bazel-out/k8-fastbuild/bin/_server.venv/lib/python3.12/site-packages/absl
lrwxrwxrwx 1 user user 68 Nov 2 18:19 ./bazel-out/k8-fastbuild/bin/_server.venv/lib/python3.12/site-packages/absl -> ../../../../../rules_python++pip+pypi_312_absl_py/site-packages/abslSetting 3 (failure ❌)
# .bazelrc
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yesDetails
$ cd ./examples/oci-container
$ bazel build //:server_layer
WARNING: Build option --@@rules_python+//python/config_settings:venvs_use_declare_symlink has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //:server_layer (0 packages loaded, 6948 targets configured).
ERROR: /home/user/bazel-contrib/rules_python/main/examples/oci-container/BUILD.bazel:20:15: Tar server_layer_default.tar.gz failed: (Exit 1): tar failed: error executing Tar command (from target //:server_layer_default) external/aspect_bazel_lib++toolchains+bsd_tar_linux_amd64/tar --create --gzip '--options=gzip:!timestamp' --file bazel-out/k8-fastbuild/bin/server_layer_default.tar.gz ... (remaining 1 argument skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
tar: Error reading archive bazel-out/k8-fastbuild/bin/server_layer.default.manifest.spec: Can't open bazel-out/k8-fastbuild/bin/_server.venv/bin/python3
tar: Error exit delayed from previous errors.
Target //:server_layer failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.836s, Critical Path: 0.35s
INFO: 5 processes: 57 action cache hit, 5 internal.
ERROR: Build did NOT complete successfully
$ readlink -f ./bazel-out/k8-fastbuild/bin/_server.venv/bin/python3 # empty
$ ls -lah ./bazel-out/k8-fastbuild/bin/_server.venv/bin/python3
lrwxrwxrwx 1 user user 78 Nov 2 18:20 ./bazel-out/k8-fastbuild/bin/_server.venv/bin/python3 -> ../../../rules_python++python+python_3_12_x86_64-unknown-linux-gnu/bin/python3Setting 4 (probably pointless, but included for completeness; failure ❌)
# .bazelrc
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=no
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yesDetails
$ cd ./examples/oci-container
$ bazel build //:server_layer
INFO: Analyzed target //:server_layer (148 packages loaded, 7018 targets configured).
ERROR: /home/user/bazel-contrib/rules_python/main/examples/oci-container/BUILD.bazel:20:15: Tar server_layer_default.tar.gz failed: (Exit 1): tar failed: error executing Tar command (from target //:server_layer_default) external/aspect_bazel_lib++toolchains+bsd_tar_linux_amd64/tar --create --gzip '--options=gzip:!timestamp' --file bazel-out/k8-fastbuild/bin/server_layer_default.tar.gz ... (remaining 1 argument skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
tar: Error reading archive bazel-out/k8-fastbuild/bin/server_layer.default.manifest.spec: Can't open bazel-out/k8-fastbuild/bin/_server.venv/bin/python3
tar: Error exit delayed from previous errors.
Target //:server_layer failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 3.157s, Critical Path: 0.66s
INFO: 46 processes: 18 internal, 28 processwrapper-sandbox.
ERROR: Build did NOT complete successfully
$ readlink -f ./bazel-out/k8-fastbuild/bin/_server.venv/bin/python3 # empty
$ ls -lah ./bazel-out/k8-fastbuild/bin/_server.venv/bin/python3
lrwxrwxrwx 1 user user 78 Nov 2 18:02 bazel-out/k8-fastbuild/bin/_server.venv/bin/python3 -> ../../../rules_python++python+python_3_12_x86_64-unknown-linux-gnu/bin/python3🌍 Your Environment
Operating System:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: nobleOutput of bazel version:
$ bazel --version
bazel 8.2.1Rules_python version:
Anything else relevant?
The issue seems to be related to how the symlinks are created in the virtual environment when venvs_site_packages is enabled. The tar command is unable to find the python3 binary in the venv. Its symlink exists on disk, as confirmed by the ls command, yet the symlink target does not, as confirmed by readlink command.