std: use readdir on nearly all UNIX platforms

[joboet]

View all comments

POSIX.1-2024 formalised what was already guaranteed by a lot of implementations and required readdir to be thread-safe as long as an individual DIR* is not accessed concurrently (which ReadDir::next ensures by taking a mutable reference). But since our read_dir implementation predates that standard, we currently only utilise readdir on the platforms that guarantee thread-safety in their documentation. On other implementations – notably macOS – we use the readdir_r function, which was always required to be thread-safe but is problematic because it cannot handle directory entries with names longer than NAME_MAX.

However, even the first POSIX issue, POSIX.1-1994, specified that the data in the returned dirent

is not overwritten by another call to readdir() on a different directory stream.

and that guarantee together with the requirement that the underlying syscalls need to be thread-safe already because of readdir_r result in readdir being thread-safe on nearly all implementations, even if they predate POSIX.1-2024. Given the now formalised guarantee I think it safe to assume that currently thread-safe implementations will not be changed in a way that violates thread-safety.

CC T-libs, do you agree?
@rustbot label +I-libs-nominated

I thus looked at the readdir implementation of all the UNIXes currently utilising readdir_r to check for thread-safety:

• Apple platforms, starting with Mac OS X 10.0. While the original implementation did not use any synchronisation, it only modifies data of the DIR* it acts on.
• Cygwin: same thing.
• Dragonfly uses locks.
• L4Re: only modifies the passed DIR*.
• LynxOS: closed-source, the documentation says that readdir is not reentrant. CC @rfatykhov-lynx
• Managarm: only modifies the passed DIR*.
• NetBSD: either uses locks depending on configuration, or only modifies the passed DIR* .
• OpenBSD: uses locks.
• Unikraft
• VxWorks: closed-source, the documentation does not make any reference to thread-safety. CC @biabbas @hax0kartik
• QNX: the documentation gives the same guarantee as POSIX.1-2024.
• NuttX: only modifies the passed DIR*.
• 3DS: the underlying directory read implementation in libctru is thread-safe
• RTEMS: I'm unable to find the implementation. CC @thesummer
• QuRT: CC @androm3da
• ESP-IDF: readdir is not thread-safe (at least on FAT) due to caching file metadata in the VFS context without locks. opendir does the same thing, too?! And the cache is never invalidated, even when the file is deleted?! Honestly, this is just broken... CC @ivmarkov @MabezDev @SergioGasquez
• Emscripten: derived from musl, only modifies the current DIR*.

On the implementations where I couldn't confirm thread-safety ReadDir will still use readdir_r, but I've changed the code so that this edge-case is limited to ReadDir::next. All other platforms now use readdir.

[rustbot]

miri is developed in its own repository. If possible, consider making this change to rust-lang/miri instead.

cc @rust-lang/miri

[rustbot]

r? @JohnTitor

rustbot has assigned @JohnTitor.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: @ChrisDenton, libs
• @ChrisDenton, libs expanded to 13 candidates
• Random selection from 7 candidates

[jonathanpallant]

@joboet - you've linked to VxWorks docs under the QNX entry?

https://www.qnx.com/developers/docs/8.0/com.qnx.doc.neutrino.lib_ref/topic/r/readdir.html says:

The readdir() function is safe to use in multithreaded environments as long as the directory stream, DIR *dirp, isn't shared between threads.

https://www.qnx.com/developers/docs/7.1/com.qnx.doc.neutrino.lib_ref/topic/r/readdir.html says the same thing

[joboet]

Whoops, I got the RTOSs confused 😉 ... thanks for the link!

[RalfJung]

Miri changes LGTM.

At some point we can probably remove readdir_r from macOS then given that it is generally considered a deprecated API, the only reason we support it is because std used it.

View changes since this review

[ivmarkov]

• ESP-IDF: readdir is not thread-safe (at least on FAT) due to caching file metadata in the VFS context without locks. opendir does the same thing, too?! And the cache is never invalidated, even when the file is deleted?! Honestly, this is just broken... CC @ivmarkov @MabezDev @SergioGasquez

@joboet I completely agree with your analysis that FAT readdir in ESP-IDF V5.4.0+ is completely broken (LittleFS and Spiffs seem OK).

I've filed the issue upstream.
In the meantime, and as per the PR, ESP-IDF should stay on readdir_r.

Thanks a lot for spotting this!

EDIT: I haven't put an effort to actually reproduce it on real hardware yet; only static analysis (confirmed by Fable though). So still a slim chance we might be missing something. Very slim. :-)

[nia-e]

Per today's t-libs meeting, we like this, +1. The possibility was also raised for those other platforms of invoking the underlying syscall directly if they're stable, but that's probably a question for a different PR. Thanks ^^

[RalfJung]

@JohnTitor reminder that this is waiting for review :)

[JohnTitor]

Sorry for the delay!
Makes sense to me, thanks for the detailed description!
r=me once the conflict is resolved

View changes since this review

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[JohnTitor]

@bors r+

[rust-bors]

📌 Commit 7f43ef4 has been approved by JohnTitor

It is now in the queue for this repository.

[Zalathar]

Possibly failed in rollup? #160469 (comment)

@bors r-
@bors try jobs=x86_64-msvc-ext2

[rust-bors]

This pull request was unapproved.

This PR was contained in a rollup (#160469), which was unapproved.

View changes since this unapproval

[rust-bors]

💔 Test for 8471650 failed: CI. Failed job:

• try - x86_64-msvc-ext2 (web logs, enhanced plaintext logs)

[rust-log-analyzer]

The job x86_64-msvc-ext2 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

tests/pass/shims/x86/rounding-error.rs ... ok
tests/pass/shims/x86/intrinsics-x86-gfni.rs ... ok

FAILED TEST: tests/pass/shims/fs.rs
command: "D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2-tools\\x86_64-pc-windows-msvc\\release\\miri.exe" "--error-format=json" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zui-testing" "--sysroot=D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\miri-sysroot" "--out-dir" "D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2-tools\\x86_64-pc-windows-msvc\\tmp\\miri_ui\\0\\tests\\pass\\shims" "tests/pass\\shims\\fs.rs" "-Zmiri-disable-isolation" "--edition" "2021" "--target" "x86_64-apple-darwin"

error: test got exit code: 1, but expected 0
 = note: compilation failed, but was expected to succeed

error: no output was expected
Execute `./miri test --bless` to update `tests/pass/shims/fs.stderr` to the actual output
+++ <stderr output>
error: unsupported operation: can't call foreign function `readdir$INODE64` on OS `macos`
##[error]  --> D:\a\rust\rust\library\std\src\sys\fs\unix.rs:865:58
   |
LL |                         let entry_ptr: *const dirent64 = readdir64(self.inner.dirp.0);
   |                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ unsupported operation occurred here
   |
   = help: this means the program tried to do something Miri does not support; it does not indicate a bug in the program
   = note: stack backtrace:
           0: <std::sys::fs::unix::ReadDir as std::iter::Iterator>::next
               at D:\a\rust\rust\library\std\src\sys\fs\unix.rs:865:58: 865:86
           1: <std::fs::ReadDir as std::iter::Iterator>::next
               at D:\a\rust\rust\library\std\src\fs.rs:2568:9: 2568:22
           2: <std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}> as std::iter::Iterator>::next
               at D:\a\rust\rust\library\core\src\iter\adapters\map.rs:107:9: 107:25
           3: <std::vec::Vec<(std::ffi::OsString, bool)> as std::vec::spec_from_iter_nested::SpecFromIterNested<(std::ffi::OsString, bool), std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}>>>::from_iter
               at D:\a\rust\rust\library\alloc\src\vec\spec_from_iter_nested.rs:24:32: 24:47
           4: <std::vec::Vec<(std::ffi::OsString, bool)> as std::vec::spec_from_iter::SpecFromIter<(std::ffi::OsString, bool), std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}>>>::from_iter
               at D:\a\rust\rust\library\alloc\src\vec\spec_from_iter.rs:33:9: 33:48
           5: <std::vec::Vec<(std::ffi::OsString, bool)> as std::iter::FromIterator<(std::ffi::OsString, bool)>>::from_iter::<std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}>>
               at D:\a\rust\rust\library\alloc\src\vec\mod.rs:3996:9: 3996:76
           6: <std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}> as std::iter::Iterator>::collect::<std::vec::Vec<(std::ffi::OsString, bool)>>
               at D:\a\rust\rust\library\core\src\iter\traits\iterator.rs:2086:9: 2086:38
           7: <std::collections::BTreeMap<std::ffi::OsString, bool> as std::iter::FromIterator<(std::ffi::OsString, bool)>>::from_iter::<std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}>>
               at D:\a\rust\rust\library\alloc\src\collections\btree\map.rs:2545:34: 2545:60
           8: <std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}> as std::iter::Iterator>::collect::<std::collections::BTreeMap<std::ffi::OsString, bool>>
               at D:\a\rust\rust\library\core\src\iter\traits\iterator.rs:2086:9: 2086:38
           9: test_directory
               at tests/pass\shims\fs.rs:373:9: 379:41
---



full stderr:
error: unsupported operation: can't call foreign function `readdir$INODE64` on OS `macos`
##[error]  --> D:\a\rust\rust\library\std\src\sys\fs\unix.rs:865:58
   |
LL |                         let entry_ptr: *const dirent64 = readdir64(self.inner.dirp.0);
   |                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ unsupported operation occurred here
   |
   = help: this means the program tried to do something Miri does not support; it does not indicate a bug in the program
   = note: stack backtrace:
           0: <std::sys::fs::unix::ReadDir as std::iter::Iterator>::next
               at D:\a\rust\rust\library\std\src\sys\fs\unix.rs:865:58: 865:86
           1: <std::fs::ReadDir as std::iter::Iterator>::next
               at D:\a\rust\rust\library\std\src\fs.rs:2568:9: 2568:22
           2: <std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}> as std::iter::Iterator>::next
               at D:\a\rust\rust\library\core\src\iter\adapters\map.rs:107:9: 107:25
           3: <std::vec::Vec<(std::ffi::OsString, bool)> as std::vec::spec_from_iter_nested::SpecFromIterNested<(std::ffi::OsString, bool), std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}>>>::from_iter
               at D:\a\rust\rust\library\alloc\src\vec\spec_from_iter_nested.rs:24:32: 24:47
           4: <std::vec::Vec<(std::ffi::OsString, bool)> as std::vec::spec_from_iter::SpecFromIter<(std::ffi::OsString, bool), std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}>>>::from_iter
               at D:\a\rust\rust\library\alloc\src\vec\spec_from_iter.rs:33:9: 33:48
           5: <std::vec::Vec<(std::ffi::OsString, bool)> as std::iter::FromIterator<(std::ffi::OsString, bool)>>::from_iter::<std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}>>
               at D:\a\rust\rust\library\alloc\src\vec\mod.rs:3996:9: 3996:76
           6: <std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}> as std::iter::Iterator>::collect::<std::vec::Vec<(std::ffi::OsString, bool)>>
               at D:\a\rust\rust\library\core\src\iter\traits\iterator.rs:2086:9: 2086:38
           7: <std::collections::BTreeMap<std::ffi::OsString, bool> as std::iter::FromIterator<(std::ffi::OsString, bool)>>::from_iter::<std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}>>
               at D:\a\rust\rust\library\alloc\src\collections\btree\map.rs:2545:34: 2545:60
           8: <std::iter::Map<std::fs::ReadDir, {closure@tests/pass\shims\fs.rs:375:18: 375:21}> as std::iter::Iterator>::collect::<std::collections::BTreeMap<std::ffi::OsString, bool>>
               at D:\a\rust\rust\library\core\src\iter\traits\iterator.rs:2086:9: 2086:38
           9: test_directory
               at tests/pass\shims\fs.rs:373:9: 379:41
---

Location:
   C:\Users\runneradmin\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\ui_test-0.30.7\src\lib.rs:365

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
error: test failed, to rerun pass `--test ui`

Caused by:
  process didn't exit successfully: `D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\build\miri/f9f9d02594f44fd6\out\ui-f9f9d02594f44fd6.exe pass` (exit code: 1)
Command `D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage0\bin\cargo.exe test --target x86_64-pc-windows-msvc -Zbinary-dep-depinfo -j 4 -Zroot-dir=D:\a\rust\rust --locked --color=always --profile=release --manifest-path D:\a\rust\rust\src/tools/miri\Cargo.toml -- pass [workdir=D:\a\rust\rust]` failed with exit code 1
Created at: src\bootstrap\src\core\build_steps\tool.rs:197:21
Executed at: src\bootstrap\src\core\build_steps\test.rs:770:19

Command has failed. Rerun with -v to see more details.
Bootstrap failed while executing `test --stage 2 src/tools/miri --target x86_64-apple-darwin --test-args pass`
Currently active steps:
test::Miri { target: x86_64-apple-darwin } at src\bootstrap\src\core\build_steps\test.rs:696
Build completed unsuccessfully in 1:30:26
  local time: Tue Aug  4 05:10:24 CUT 2026

[RalfJung]

Apparently Miri tests are failing on x86_64-apple-darwin. PR CI passed, meaning aarch64-apple-darwin seems okay... I did not realize that those targets sometimes use different symbols.