Cannot install Pants 2.31.0 due to "Failed to establish atomic directory"

[nikhilxb]

Describe the bug
Cannot start Pants from launcher binary ~/.local/bin/pants.

Pants version
2.31.0

OS
Linux

Operating System: Red Hat Enterprise Linux 9.6 (Plow)     
     CPE OS Name: cpe:/o:redhat:enterprise_linux:9::baseos
          Kernel: Linux 5.14.0-570.12.1.el9_6.x86_64

Additional info

Running the following after clean install after curl install script, from within repo with pants.toml and .pants.bootstrap.

> pants version
Downloading https://github.com/astral-sh/python-build-standalone/releases/download/20250612/cpython-3.9.23%2B20250612-x86_
[...]
Downloading https://github.com/astral-sh/python-build-standalone/releases/download/20250612/cpython-3.9.23%2B20250612-x86_
Traceback (most recent call last):
  File "[$SCRATCH]/cache/scie/[HASH1]/cpython-3.9.23+20250612-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "[$SCRATCH]cache/scie/[HASH1]/cpython-3.9.23+20250612-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "[$SCRATCH]/cache/scie/[HASH2]/tools.pex/__main__.py", line 233, in <module>
  File "[$SCRATCH]/cache/scie/[HASH2]/tools.pex/__main__.py", line 213, in boot
  File "[$SCRATCH]/cache/scie/[HASH2]/tools.pex/__main__.py", line 93, in __ensure_pex_installed__
  File "[$SCRATCH]/cache/scie/[HASH2]/tools.pex/.bootstrap/pex/layout.py", line 696, in ensure_installed
  File "[$SCRATCH]/cache/scie/[HASH2]/tools.pex/.bootstrap/pex/layout.py", line 326, in _ensure_installed
  File "[$SCRATCH]/cache/scie/[HASH2]/tools.pex/.bootstrap/pex/cache/access.py", line 91, in read_write
  File "[$SCRATCH]/cache/scie/[HASH2]/tools.pex/.bootstrap/pex/cache/access.py", line 79, in _lock
OSError: [Errno 9] Bad file descriptor
Error: Failed to establish atomic directory [$SCRATCH]/cache/scie/[HASH3]/locks/configure-[HASH4]. Population of work directory failed: Boot binding command failed: exit status: 1

Isolates your Pants from the elements.

Please select from the following boot commands:

<default> (when SCIE_BOOT is not set in the environment)  Detects the current Pants installation and launches it.
bootstrap-tools                                           Introspection tools for the Pants bootstrap process.
update                                                    Update scie-pants.

You can select a boot command by setting the SCIE_BOOT environment variable.

Contents of pants.toml:

[GLOBAL]
pants_version = "2.31.0"

backend_packages = [
  "pants.backend.python",
]

Contents of .pants.bootstrap:

export SCIE_BASE=$SCRATCH/cache/scie
export SCIE_CACHE=$SCRATCH/cache/nce
export PEX_ROOT=$SCRATCH/cache/pex
mkdir -p $SCRATCH/cache/scie $SCRATCH/cache/nce $SCRATCH/cache/pex

This doesn't work even when running with the default cache root directories (i.e. without .pants.bootstrap). In case it's relevant, I am working on a HPC cluster.

[jsirois]

@nikhilxb what filesystems are in play? None of Pants, Pex or scies work on network mounts.

The code failing in this case is in Pex here: https://github.com/pex-tool/pex/blob/1eb83f643f42dd0868420e7de4e57fffb5444ae5/pex/cache/access.py#L76-L79

[nikhilxb]

@jsirois Thanks for the quick reply. I’m running this on an HPC cluster with network-mounted filesystems, so that probably explains the error. I hadn’t realized from reading the docs that network mounts could be an issue here. I'm a bit surprised tbh since I’d assume it's common for larger/cloud monorepos to span diverse and mounted filesystems — but maybe I'm missing something here about why it doesn't just work.

Is there a way to work around this constraint? I don’t necessarily need PEX binaries/targets; I'd be happy to have Pants run on a local Python interpreter or build some kind of local ./pants script.

Happy to share more information about the cluster configuration — just let me know what commands would be useful to run!

[jsirois]

The constraint comes from at least three independent sources:

1. Pex cache locks: use flock and fcntl
2. Scie cache locks: same
3. Pants LMBD use.

So there is no way around this. Even if you strip away 1 & 2 you're still left with 3, and LMDB is central to Pants itself. I'll let Pants maintainers explain more about that, but you can at least read here: http://www.lmdb.tech/doc/ - search for "remote filesystems" and find:

Do not use LMDB databases on remote filesystems, even between processes on the same host. This breaks flock() on some OSes, possibly memory map sync, and certainly sync between programs on different hosts.

[nikhilxb]

@jsirois I see … Could source 3 (Pants LMDB) be avoided by passing --no-local-cache, aka local_cache = False in pants.toml?

From Pants docs at https://www.pantsbuild.org/stable/docs/writing-plugins/the-rules-api/tips-and-debugging#fyi-caching-semantics:

There are two layers to Pants caching: in-memory memoization and caching written to disk via the LMDB store.
Pants will write to the LMDB store—usually at ~/.cache/pants/lmdb_store—for any Process execution and when "digesting" files. […] The cache will be occasionally garbage collected by Pantsd, and users may also use --no-local-cache.

[jsirois]

@nikhilxb perhaps, but at that point you're not using Pants! You've knee-capped it.

Does your HPC cluster actually have 0 local disk for you to use? HPC makes sense for actual high performance computing, but for building? That makes less sense. Build tools often support distributed builds - distcc, Pants with remoting enabled, etc - but building in an HPC cluster - what are you aiming for?

[nikhilxb]

@jsirois Haha maybe, but we would rather run a knee-capped Pants than no Pants at all!

Glad to hear that constraint 3 is not a hard constraint. Surely the --no-local-cache flag is there for a reason, and the cache is periodically deleted anyways.

To answer your question about use case: HPC clusters are the main way that most academic institutions I know of share resources. We develop "in the cloud" with each user given a part of the shared filesystem, then submit jobs to dedicated compute (e.g. via SLURM). My collaborators and I all perform development on this system, and we're looking to adopt Pants for 2 main reasons:

1. To abstract away our tooling (formatters, etc.) under a common CLI interface. This improves the devex particularly for newer developers who are unfamiliar with all the necessary commands.
2. To perform dependency inference. We are doing AI research, and it is very common to need to publish open-source repos by exporting a slice of our monorepo. We current do this manually which is tedious and error-prone. It'll be much easier to use the python_library target and export all transitive dependencies.

Following Pants's incremental adoption philosophy, if things went well we were open to more complex build targets like Docker or PyPI packaging. But for now, we certainly don't need any of the complex remote building capabilities you alluded to.

We don't need Pants to be optimized, but we would like to install it at least, even if it means circumventing the remaining constraint sources 1 and 2 (Pex cache locks, SCIE cache locks) by any means necessary.

[jsirois]

Surely the --no-local-cache flag is there for a reason

Yes, mainly debugging!. I encourage you to get Pants maintainers involved. Perhaps ping them on Slack, they are often sluggish dealing with the issue channel here. You truly are trying to do something novel. I think it's misguided, but you really need someone with a stake in Pants to help guide you here.

[cburroughs]

hi @nikhilxb I wanted to clarify where you are trying to eventually get. Are you trying to:

• (A) Install Pants-the-binary on a network-mounted filesystems; ~/.cache, the source code, etc would be on local workstations. (You can assume one "admin" writer, and everyone else is read-only.)
• (B) Run and use Pants when /home/$USER is a network mount
• (C) Run and use Pants where ~/.cache or the source code are a shared read/write network mount?

[nikhilxb]

@cburroughs Thanks for your help! If I understand you correctly, it'd be (B) and (C).

/home/$USER is a network mount. /home/$USER/monorepo is the source repo that we want to run Pants in.

We don't really care if there's a cache (that's up to the implementations of Pex/Scie/Pants). But if it must exist then it could be placed in /scratch/$USER (another network mount; preferred over /home due to org-specific inode quota restrictions) or some sort of /tmp filesystem that is cleared regularly.

We are also completely fine with not using PEX, and just forcing Pants to use a local Python interpreter (e.g. /usr/bin/python or /home/$USER/python). It seems from looking at the docs that previous versions maybe used a local ./pants launcher script?

Pants's dependency inference and extensible tool abstraction would be very useful for us, and unfortunately these features are difficult to replicate by other means. I can imagine many people on shared computing clusters that would appreciate its benefits.

[cburroughs]

Thanks, that helps a lot. I've noodled a bit and I'm trying to balance "I think this use case is cool and interesting" with "I think this is pretty far from what the project supports today". I should say that I haven't had my home directory on an NFS mount for > 20 years.

As John outlined, there are several different levels at play:

• scie/pex for installation (use flock and fcntl). At the end of the day these are self extracting archives, and I think it is likely there would be some path to extracting them to a network mount in a "one writer:many readers" model (As you alluded to, Pants in this repo uses a 'regular' Python and that is still possible.)
• Pex cache (use flock and fcntl): These are fundamental to most everything Pants does today with Python. The model here is that data flows from Pex's cache into execution sandbox, not as an optional component that could be /dev/null. You didn't mention what language you wanted to code in but I'm guessing there is at least some Python.
• Pants foundational LMDB cache with the "Do not use LMDB databases on remote filesystems," warning. In an ideal world Pants without this cache would be fast, and with it super fast. But in practice it is almost certainly needed for adequate performance.

How this all works in non-HPC land is that Pants uses the local workstation filesystem for all of those, and pushes artifacts (Docker, tarballs, Pex, etc) to some networked service for sharing or deployment. I hope that helps. Let me know if there is more I can shed light on that would assist you with your exploration.

[jsirois]

For cross reference, the existing: pantsbuild/scie-pants#483

[nikhilxb]

@cburroughs @jsirois Okay, I've found a workaround!

It involves installing Pants and all caches on the /tmp directory (which is non-NFS), then symlinking a couple directories to get around SCIE/PEX apparently still reading from ~/.cache in some cases, perhaps due to hardcoding paths.

Workaround

1. Search for non-NFS filesystems on the HPC compute nodes. We found that /tmp was XFS, while our $HOME and $SCRATCH directories were NFS.

> df -T /tmp $HOME $SCRATCH
Filesystem    Type     1K-blocks          Used     Available Use% Mounted on
[NAME1]       xfs       81854464      11150920      70703544  14% /
[NAME2]       nfs4  976562500608   14932824064  961629676544   2% /home/[$USER]
[NAME3]       nfs4 2929687500800 1628337870848 1301349629952  56% /scratch/[$USER]

2. Configure the Pants bootstrap script and config toml to create caches on /tmp.

~/monorepo/.pants.bootstrap:

# Relocate cache directories to non-NFS filesystem.
export SCIE_BASE=/tmp/$USER/scie
export SCIE_CACHE=/tmp/$USER/nce
export PEX_ROOT=/tmp/$USER/pex
export PANTS_SETUP_CACHE=/tmp/$USER/pants/setup

# Create cache directories if they don't exist.
mkdir -p $SCIE_BASE $SCIE_CACHE $PEX_ROOT $PANTS_SETUP_CACHE

# Create symbolic links from ~/.cache to circumvent 'nce'/'pex' bugs.
ln -sf $SCIE_CACHE ~/.cache/nce
ln -sf $PEX_ROOT ~/.cache/pex

~/monorepo/pants.toml:

[GLOBAL]
pants_version = "2.31.0"

# Relocate cache directories to non-NFS filesystem.
local_store_dir = "/tmp/%(env.USER)s/pants/lmdb_store"
named_caches_dir = "/tmp/%(env.USER)s/pants/named_caches"

3. Pants now works when running pants ... (from ~/.local/bin/pants).

Additional Info

After configuring just the variables in the script and toml files, Python now successfully installs within /tmp/$USER/scie/.... It no longer errors during download as in the original message. However, it now errors with the same message at a later stage:

> pants 
Bootstrapping Pants 2.31.0
Installing pantsbuild.pants==2.31.0 into a virtual environment at /tmp/[$USER]/scie/[HASH1]/bindings/venvs/2.31.0
Failed to create Pants virtual environment.
Error: Command '['/tmp/[$USER]/scie/[HASH1]/bindings/pex_root/venvs/1/[HASH2]/[HASH3]/bin/python', '/tmp/tmpxapafqv_.pex', 'venv', '--prompt', 'Pants 2.31.0', '--compile', '--pip', '--collisions-ok', '--no-emit-warnings', '--disable-cache', '/tmp/[$USER]/scie/[HASH1]/bindings/venvs/2.31.0']' returned non-zero exit status 1., output:
-----
b'Traceback (most recent call last):\n  File "<frozen runpy>", line 198, in _run_module_as_main\n  File "<frozen runpy>", line 88, in _run_code\n  File "/tmp/tmpxapafqv_.pex/__main__.py", line 244, in <module>\n  File "/tmp/tmpxapafqv_.pex/__main__.py", line 224, in boot\n  File "/tmp/tmpxapafqv_.pex/__main__.py", line 95, in __ensure_pex_installed__\n  File "/tmp/tmpxapafqv_.pex/.bootstrap/pex/layout.py", line 712, in ensure_installed\n  File "/tmp/tmpxapafqv_.pex/.bootstrap/pex/layout.py", line 332, in _ensure_installed\n  File "/tmp/tmpxapafqv_.pex/.bootstrap/pex/cache/access.py", line 86, in read_write\n  File "/tmp/tmpxapafqv_.pex/.bootstrap/pex/cache/access.py", line 75, in _lock\n  File "/tmp/tmpxapafqv_.pex/.bootstrap/pex/fs/lock.py", line 73, in acquire\n  File "/tmp/tmpxapafqv_.pex/.bootstrap/pex/fs/_posix.py", line 34, in acquire\nOSError: [Errno 9] Bad file descriptor\n'
-----

Error: Failed to establish atomic directory /tmp/[$USER]/scie/[HASH1]/locks/install-[HASH4]. Population of work directory failed: Boot binding command failed: exit status: 1

In the process of this failed run, it creates ~/.cache/nce and ~/.cache/pex, in particular the following 3 files:

> ls ~/.cache/nce/[HASH]/
scie-pants.bin  scie-pants.lck  
> ls ~/.cache/pex/
access.lck 

Thus, it is ignoring our config variables to use the original cache paths. We circumvent this by symlinking the directories to /tmp/$USER/nce and /tmp/$USER/pex. After this, Pants seems to work: we ran the help, count-loc, and tailor goals successfully.

Please let me know if you see any obvious problems with this! Or if there's a more elegant solution than symlinking (e.g. another config variable we're not setting)?

[jsirois]

@nikhilxb great.

Does your HPC cluster actually have 0 local disk for you to use?

Ok, so you did find local disk. Watch out for /tmp cleaners - which are often a thing. Perhaps you've already looked into this and determined you're safe or don't mind /tmp/ being cleaned out from under you.

then symlinking a couple directories to get around SCIE/PEX apparently still reading from ~/.cache in some cases, perhaps due to hardcoding paths.

So, here's a ~minimal config. N.B. I'm leveraging the fact that scies support loading .env files early ¹ and scie-pants leverages this (see: https://github.com/pantsbuild/scie-pants/tree/main?tab=readme-ov-file#features); so the env export is set up in .env (to ensure scie-pants sees those env vars) and then .pants.bootstrap just sources that file to make double-sure pants sees those env vars. Further, instead of editing pants.toml like you did I just export the equivalent PANTS_* env vars:

:; cat Dockerfile 
FROM ubuntu:24.04

RUN apt update && \
    DEBIAN_FRONTEND=noninteractive apt upgrade --yes && \
    DEBIAN_FRONTEND=noninteractive apt install --yes \
        curl \
        git \
        vim

ENV USER=root PATH=/tmp/bin:$PATH

RUN git clone https://github.com/pantsbuild/example-python /tmp/example-python && \
    echo 'export SCIE_BASE=/tmp/$USER/nce' >> /tmp/example-python/.env && \
    echo 'export PEX_ROOT=/tmp/$USER/pex' >> /tmp/example-python/.env && \
    echo 'export PANTS_LOCAL_STORE_DIR=/tmp/$USER/pants/lmdb_store' >> /tmp/example-python/.env && \
    echo 'export PANTS_NAMED_CACHES_DIR=/tmp/$USER/pants/named_caches' >> /tmp/example-python/.env && \
    echo 'export PANTS_SETUP_CACHE=/tmp/$USER/pants/setup' >> /tmp/example-python/.env && \
    echo 'source .env' >> /tmp/example-python/.pants.bootstrap

RUN curl --proto '=https' --tlsv1.2 -fsSL https://static.pantsbuild.org/setup/get-pants.sh | \
    bash -s -- --bin-dir /tmp/bin

WORKDIR /tmp/example-python

:; docker build . -t pants-issues-23129 -q
sha256:1e7dd64a42b6b6f27e63ceca117646e92c3f05ce9a19774cd9c5916871885110
:; docker run --rm -it pants-issues-23129
root@9b1d2ed9711c:/tmp/example-python# ls ~/.cache
ls: cannot access '/root/.cache': No such file or directory
root@9b1d2ed9711c:/tmp/example-python# pants -V
Bootstrapping Pants 2.31.0
Installing pantsbuild.pants==2.31.0 into a virtual environment at /tmp/root/nce/5c54a0c25255df2afb0525da856d49eb81b48cff0e32a9c868f77a296af24319/bindings/venvs/2.31.0
New virtual environment successfully created at /tmp/root/nce/5c54a0c25255df2afb0525da856d49eb81b48cff0e32a9c868f77a296af24319/bindings/venvs/2.31.0
23:14:46.02 [INFO] Initializing scheduler...
23:14:46.12 [INFO] Scheduler initialized.
2.31.0
root@9b1d2ed9711c:/tmp/example-python# ls ~/.cache/
pex
root@9b1d2ed9711c:/tmp/example-python# ls -1 ~/.cache/pex/unzipped_pexes/3/bcdb98ffefa721203dc7dff0ac7635de6fd537d3/.deps/
ansicolors-1.1.8-py2.py3-none-any.whl
chevron-0.14.0-py3-none-any.whl
fasteners-0.20-py3-none-any.whl
hdrhistogram-0.10.3-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
ijson-3.4.0.post0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
libcst-1.8.5-cp311-cp311-manylinux_2_28_x86_64.whl
node_semver-0.9.0-py3-none-any.whl
packaging-25.0-py3-none-any.whl
pantsbuild_pants-2.31.0-cp311-cp311-linux_x86_64.whl
pbr-7.0.3-py2.py3-none-any.whl
psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
python_lsp_jsonrpc-1.1.2-py3-none-any.whl
pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
setproctitle-1.3.7-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl
setuptools-80.9.0-py3-none-any.whl
toml-0.10.2-py2.py3-none-any.whl
types_pyyaml-6.0.12.20250915-py3-none-any.whl
types_toml-0.10.8.20240310-py3-none-any.whl
typing_extensions-4.15.0-py3-none-any.whl
ujson-5.11.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Now those deps in the ~/.cache/pex root look an awful lot like Pants own deps; so that pollution is coming from the install of the Pants PEX. That happens here:
https://github.com/pantsbuild/scie-pants/blob/3203faa30d46800dadba485edd1167246c1dde3a/tools/src/scie_pants/install_pants.py#L74-L112

Look at line 108. That's a scie-pants bug. It's ignoring all env except the PEX_TOOLS=1 it sets, and so Pex uses the default PEX_ROOT as it should, which is ~/.cache/pex on Linux. This was a bug from day 1 of scie-pants switching from installing Pants from a .whl to installing Pants from a PEX here: https://github.com/pantsbuild/scie-pants/pull/226/changes#r2892914835

So, and this is being pedantic, but I'm the maintainer of both scie and Pex, the bug here is in scie-pants, not scie and not Pex. I'll leave it to you or @cburroughs or @sureshjoshi (who is working in this area lately) to fix that bug.

Footnotes

1. https://github.com/pantsbuild/scie-pants/blob/3203faa30d46800dadba485edd1167246c1dde3a/package/scie-pants.toml#L1-L4 ↩