Use futex-based synchronization on Apple platforms #122408

joboet · 2024-03-12T20:36:51Z

Last week, Apple released macOS version 14.4, which introduced a public futex API called os_sync_wait_on_address (Apple has failed to include the documentation provided in the defining header os/os_sync_wait_on_address.h on its website). As the private API backing these functions has been around since macOS 10.12, our minimum supported version, it can be used as fallback without risking breakage in future versions.

This PR thus switches all other synchronization primitives except Mutex (namely Condvar, RwLock, Once and thread parking) to the futex-based implementations also used on Linux and Windows.

rustbot · 2024-03-12T20:36:58Z

r? @m-ou-se

rustbot has assigned @m-ou-se.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

library/std/src/sys/pal/unix/futex.rs

BlackHoleFox · 2024-03-13T01:29:57Z

library/std/src/sys/pal/unix/futex.rs

+
+    // These syscalls appeared with macOS 10.12.
+    weak! {
+        pub fn __ulock_wait(u32, *mut c_void, u64, u32) -> c_int


Mind explaining more what the thought process of these going undetected is? Seems like a risky chance that might require a hot fix if the MAS or iOS app store scanners see through this.

In reality I think this won't be enough and we need to call ulock things via syscall directly instead. This means that the "fallback" Apple futex implementation is only going to be usable on macOS.

I know Electron apps on the MAS use direct syscalls to emulate private functions, so this is probably what Rust should do too on macOS with precedence:

#define SYS_ulock_wait 515 #define SYS_ulock_wake 516 #define SYS_ulock_wait2 544

For iOS, detect if the os_sync_ functions are available or fallback to the non-futex implementation :/ Calling a syscall directly is forbidden per the iOS headers so std can't do the same as macOS:

__WATCHOS_PROHIBITED __TVOS_PROHIBITED __OS_AVAILABILITY_MSG(ios,deprecated=10.0,"syscall(2) is unsupported; " "please switch to a supported interface. For SYS_kdebug_trace use kdebug_signpost().") __OS_AVAILABILITY_MSG(macosx,deprecated=10.12,"syscall(2) is unsupported; " "please switch to a supported interface. For SYS_kdebug_trace use kdebug_signpost().") int syscall(int, ...);

This is a "spirit of the law" kind of situation. Apple understandably wants App Store software to continue working in newer versions, so they prohibit private API use, as they want to be able to remove or change that API at will. By only using the private API as fallback and not linking it directly, we fulfil this wish. If we directly linked the private API instead, we'd get dynamic linker errors if they remove it, so we can't do that.

On the other hand, we break the "letter of the law" as we are using private API. In my opinion, this is totally fine and justified, but of course, this is only my interpretation. It would be great if someone from Apple could look over this, just so that we know that they are aware of this. My past attempt at reaching out to them for this has been unsuccessful.

Would you consider the solution of atomic-wait? They use libc++.

https://github.com/m-ou-se/atomic-wait/blob/main/src/macos.rs

But these only exist starting with macOS 11, which is way above our minimum of 10.12, so the ulock fallback would be needed regardless.

This sets precedent for using private APIs in the standard library (our current use of weak! is only used for symbols that are available in newer versions), which I think is a fairly big deal, and should perhaps be discussed more broadly (in a separate issue? (that could be FCP'ed?)).

Somewhat related is #114186. Tagging in particular @thomcc and @workingjubilee, as they seem to have been involved in this kind of stuff before?

At the very least, we should use the dlsym! macro explicitly here, to make it very clear that we're not weakly linking the symbol (which would be very visible in the binary), but actually loading it at runtime, and thereby evading the App Store's checks (dlsym itself seems to be allowed).

I believe this is sufficient such that we do not need to do raw syscalls (which I remember to have caused problems for Go in the past since as the ABI isn't stable (?)).

Yeah, we shouldn't be calling this directly, it'll trip the app store static analysis.

BlackHoleFox · 2024-03-13T01:30:23Z

library/std/src/sys/pal/unix/futex.rs

+                wait(UL_COMPARE_AND_WAIT | ULF_NO_ERRNO, addr, value, 0);
+            }
+        } else {
+            panic!("your system is below the minimum supported version of Rust");


These panics also get to be deleted if raw syscalls are used :)

If we do keep the panics, I'd go for an abort here instead, this is not recoverable in any way and avoiding the overhead from unwind landing pads would be nice.

bors · 2024-03-13T10:15:08Z

☔ The latest upstream changes (presumably #122423) made this pull request unmergeable. Please resolve the merge conflicts.

Implement the `os_unfair_lock` functions on macOS These are needed for rust-lang/rust#122408. See the documentation [here](https://developer.apple.com/documentation/os/synchronization?language=objc) and the implementation [here](https://github.com/apple-oss-distributions/libplatform/blob/a00a4cc36da2110578bcf3b8eeeeb93dcc7f4e11/src/os/lock.c#L645).

Implement the `os_unfair_lock` functions on macOS These are needed for rust-lang#122408. See the documentation [here](https://developer.apple.com/documentation/os/synchronization?language=objc) and the implementation [here](https://github.com/apple-oss-distributions/libplatform/blob/a00a4cc36da2110578bcf3b8eeeeb93dcc7f4e11/src/os/lock.c#L645).

madsmtm

In general, nice work, this will be a great perf improvement!

I know it is a draft, but once you're ready, it'd be nice to split the Mutex changes apart from the futex changes, if possible.

@rustbot label O-apple

madsmtm · 2024-09-26T12:21:40Z

library/std/src/sys/locks/condvar/mod.rs

+        target_os = "macos",
+        target_os = "ios",
+        target_os = "tvos",
+        target_os = "watchos",


Please use target_vendor = "apple", as that also includes visionOS. Also applies elsewhere.

library/std/src/sys/locks/mutex/apple.rs

madsmtm · 2024-09-26T13:01:50Z

library/std/src/sys/pal/unix/futex.rs

+/// See os/os_sync_wait_on_address.h (Apple has failed to upload the documentation
+/// to their website) for documentation of the public API and
+/// https://github.com/apple-oss-distributions/xnu/blob/1031c584a5e37aff177559b9f69dbd3c8c3fd30a/bsd/sys/ulock.h#L69
+/// for the header file of the private API, along with its usage in libpthread
+/// https://github.com/apple-oss-distributions/libpthread/blob/d8c4e3c212553d3e0f5d76bb7d45a8acd61302dc/src/pthread_cond.c#L463


I'd also link the design/decision document:
https://github.com/apple-oss-distributions/libplatform/blob/libplatform-316.100.10/docs/expose-futex-style-api.md

library/std/src/sys/pal/unix/futex.rs

madsmtm · 2024-09-26T13:39:47Z

library/std/src/sys/pal/unix/futex.rs

+    if let Some(wake) = os_sync_wake_by_address_any.get() {
+        unsafe { wake(addr, size_of::<u32>(), OS_SYNC_WAKE_BY_ADDRESS_NONE) == 0 }
+    } else if let Some(wake) = __ulock_wake.get() {
+        // __ulock_wake can get interrupted, so retry until either waking up a
+        // waiter or failing because there are no waiters (ENOENT).
+        loop {
+            let r = unsafe { wake(UL_COMPARE_AND_WAIT | ULF_NO_ERRNO, addr, 0) };
+
+            if r >= 0 {
+                return true;
+            } else {
+                match -r {
+                    libc::ENOENT => return false,
+                    libc::EINTR => continue,
+                    err => panic!("__ulock_wake failed: {}", Error::from_raw_os_error(err)),
+                }
+            }
+        }


Hmm, are you sure this is correct? The implementation of os_sync_wake_by_address_any seems to just call __ulock_wake without a loop? So either we need to loop both places, or in none of them?

Same goes for futex_wake.

the8472 · 2024-10-23T17:18:16Z

We discussed this during today's T-libs meeting (albeit with low attendance). We might be willing to go along with the changes on the theory that apps are validated on current OS versions and will use current stable APIs and so shouldn't cause issues and that the fallback APIs have been available for a long time and so won't suddenly disappear on older versions.

But in previous cases where we've started to rely on internal APIs we've leaned on platform experts and their investigations how risky that would be. So we'd like to do the same here. @rustbot ping apple

rustbot · 2024-10-23T17:18:19Z

Hey Apple notification group! This issue or PR could use some Apple-specific
guidance. Could one of you weigh in? Thanks <3

(In case it's useful, here are some instructions for tackling these sorts of
issues).

cc @BlackHoleFox @hkratz @inflation @madsmtm @nvzqz @shepmaster @thomcc

the8472 · 2024-10-23T17:26:55Z

This sets precedent for using private APIs in the standard library (our current use of weak! is only used for symbols that are available in newer versions), which I think is a fairly big deal, and should perhaps be discussed more broadly (in a separate issue? (that could be FCP'ed?)).

We do use internal APIs on windows (NtCreateFile), but that became necessary to fix a CVE and was used after looking at the microsoft ecosystem official projects. And windows is a more open platform in a way (hah) since they don't review most software. So it's not fully comparable.

madsmtm · 2024-10-23T21:36:38Z

on the theory that apps are validated on current OS versions and will use current stable APIs and so shouldn't cause issues and that the fallback APIs have been available for a long time and so won't suddenly disappear on older versions.

Just to be clear, the problematic API here is not so much the newly added APIs (os_sync_wait_on_address etc.), but the fallback implementation for older OS'es (__ulock_wait) that might trip Apple's analysis, and make it very hard for Rust users to submit their application to the App Store.

It would be nice if someone could try to submit an application for review in the App Store that uses this branch of rustc, and see if Apple's review picks up on the usage of __ulock_wait via. dlsym or not, that would appease most of my worries here.

Shamelessly stolen from pending PR: rust-lang/rust#122408

This is necessary to unblock rust-lang/rust#122408. The documentation for these is available [here](https://developer.apple.com/documentation/os/os_sync_wait_on_address?language=objc). Because the futex wait operations (`os_sync_wait_on_address` et al.) return the number of remaining waiters after returning, this required some changes to the common futex infrastructure, which I've changed to take a callback instead of precalculating the return values.

This is necessary to unblock rust-lang#122408. The documentation for these is available [here](https://developer.apple.com/documentation/os/os_sync_wait_on_address?language=objc). Because the futex wait operations (`os_sync_wait_on_address` et al.) return the number of remaining waiters after returning, this required some changes to the common futex infrastructure, which I've changed to take a callback instead of precalculating the return values.

Miri subtree update r? `@ghost` Unblocks rust-lang#122408 from the Miri side

Rollup merge of rust-lang#136452 - RalfJung:miri-sync, r=RalfJung Miri subtree update r? `@ghost` Unblocks rust-lang#122408 from the Miri side

Miri subtree update r? `@ghost` Unblocks rust-lang/rust#122408 from the Miri side

joboet · 2025-02-03T15:51:54Z

I've removed the Mutex change, so this only contains the switch to futex-based implementations. With rust-lang/miri#4142 merged, this should be ready for review.

@rustbot ready

joboet · 2025-02-03T15:54:05Z

As for the whole App Store situation: I consider it highly unlikely that Apple's automatic scanners will detect our usage of dlsym – they would either need to scan every string in the binary or run Apps on older platforms, both of which seem unlikely.

@rustbot label +I-libs-nominated

m-ou-se · 2025-02-06T12:53:33Z

As for the whole App Store situation: I consider it highly unlikely that Apple's automatic scanners will detect our usage of dlsym – they would either need to scan every string in the binary or run Apps on older platforms, both of which seem unlikely.

We discussed this in the libs meeting. We agree that your approach is low risk, and are happy to try this out.

bors · 2025-02-10T14:47:21Z

☔ The latest upstream changes (presumably #136809) made this pull request unmergeable. Please resolve the merge conflicts.

BlackHoleFox · 2025-02-10T23:40:02Z

... they would either need to scan every string in the binary or run Apps on older platforms, both of which seem unlikely.

I don't think historical evidence backs you up, whatever its worth @joboet. Just as a random example, Python 3.12 started getting app's rejected because a full-text string scan of binaries contained a forbidden string name: https://lwn.net/Articles/979671/. Quoting from one of the related threads:

The submission process is entirely inscrutable; but as far as it is possible to work out, it’s a literal substring match on the py/pyc file. And yes, it is awful, in every sense of the word.

I'm also aware of several Mac apps which obfuscate strings in their builds to avoid the scanner problem as well, and my background radiation knowledge is that string scanning combined with symbol scanning is how a chunk of app store private API checks are done.

Tis' the lib team's choice at the end of the day though and its not like temporarily not being able to upgrade Rust in an app store distributed project is the end of the world.

Miri subtree update r? `@ghost` Unblocks rust-lang#122408 from the Miri side

RalfJung · 2025-08-26T18:00:04Z

So, uh, what should we do here? Roll this out and risk a revert? Wait until all supported versions supported Futexes? Obfuscate the string passed to dlsym?

madsmtm · 2025-08-26T19:30:10Z

I've been talking with a friend who has an app in the App Store, and got him to submit a few different versions of his app using various private symbols in different ways as a test (basically probing the black box), still waiting on the last result, give me a few weeks.

joboet added the A-atomic Area: Atomics, barriers, and sync primitives label Mar 12, 2024

joboet marked this pull request as draft March 12, 2024 20:36

rustbot assigned m-ou-se Mar 12, 2024

rustbot added O-unix Operating system: Unix-like S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Mar 12, 2024

joboet mentioned this pull request Mar 12, 2024

Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740

Closed

63 tasks

BlackHoleFox suggested changes Mar 13, 2024

View reviewed changes

joboet added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 8, 2024

RalfJung mentioned this pull request Apr 18, 2024

Implement macOS dispatch semaphores rust-lang/miri#2589

Open

This was referenced Jul 9, 2024

macOS: add os_sync_wait_on_address and related definitions rust-lang/libc#3769

Merged

Implement the os_unfair_lock functions on macOS rust-lang/miri#3745

Merged

madsmtm requested changes Sep 26, 2024

View reviewed changes

rustbot added O-apple Operating system: Apple (macOS, iOS, tvOS, visionOS, watchOS) labels Sep 26, 2024

RalfJung mentioned this pull request Sep 29, 2024

Can we avoid the heap allocation in macOS Mutex/Condvar? #131005

Open

joboet mentioned this pull request Oct 10, 2024

Shim the macOS futex API rust-lang/miri#3962

Closed

joboet added the I-libs-nominated Nominated for discussion during a libs team meeting. label Oct 10, 2024

Dylan-DPC added S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Oct 14, 2024

joshtriplett added the relnotes Marks issues that should be documented in the release notes of the next release. label Nov 6, 2024

Grimeh added a commit to Grimeh/atomic-wait that referenced this pull request Dec 1, 2024

Add macos futex impl

40906de

Shamelessly stolen from pending PR: rust-lang/rust#122408

madsmtm mentioned this pull request Jan 5, 2025

RFC: cfg_target_version rust-lang/rfcs#3750

Open

joboet mentioned this pull request Jan 19, 2025

shim Apple's futex primitives rust-lang/miri#4142

Merged

madsmtm mentioned this pull request Jan 21, 2025

Alacritty hangs occasionally on startup alacritty/alacritty#8444

Closed

RalfJung mentioned this pull request Feb 2, 2025

Miri subtree update #136452

Merged

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Feb 2, 2025

Rollup merge of rust-lang#136452 - RalfJung:miri-sync, r=RalfJung

a8055f9

Miri subtree update r? `@ghost` Unblocks rust-lang#122408 from the Miri side

bors added a commit to rust-lang-ci/rust that referenced this pull request Feb 2, 2025

Auto merge of rust-lang#136452 - RalfJung:miri-sync, r=RalfJung

af3c00d

Miri subtree update r? `@ghost` Unblocks rust-lang#122408 from the Miri side

rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Feb 3, 2025

Unrolled build for rust-lang#136452

b615b9a

Rollup merge of rust-lang#136452 - RalfJung:miri-sync, r=RalfJung Miri subtree update r? `@ghost` Unblocks rust-lang#122408 from the Miri side

github-actions bot pushed a commit to rust-lang/rustc-dev-guide that referenced this pull request Feb 3, 2025

Rollup merge of #136452 - RalfJung:miri-sync, r=RalfJung

cce8d5c

Miri subtree update r? `@ghost` Unblocks rust-lang/rust#122408 from the Miri side

github-actions bot pushed a commit to rust-lang/miri that referenced this pull request Feb 3, 2025

Rollup merge of #136452 - RalfJung:miri-sync, r=RalfJung

832d641

Miri subtree update r? `@ghost` Unblocks rust-lang/rust#122408 from the Miri side

std: allow weak in item position on Apple platforms

c9f7324

joboet force-pushed the apple_futex branch from 76b3461 to 7af895d Compare February 3, 2025 15:50

joboet marked this pull request as ready for review February 3, 2025 15:51

rustbot added the I-libs-nominated Nominated for discussion during a libs team meeting. label Feb 3, 2025

std: use futex for everything but Mutex on Apple platforms

dc49e21

joboet force-pushed the apple_futex branch from 7af895d to dc49e21 Compare February 5, 2025 09:03

m-ou-se removed the I-libs-nominated Nominated for discussion during a libs team meeting. label Feb 5, 2025

madsmtm mentioned this pull request Feb 11, 2025

Implement cfg_os_version_min #136867

Draft

github-actions bot pushed a commit to tautschnig/verify-rust-std that referenced this pull request Mar 11, 2025

Rollup merge of rust-lang#136452 - RalfJung:miri-sync, r=RalfJung

7d6cbfd

Miri subtree update r? `@ghost` Unblocks rust-lang#122408 from the Miri side

oxalica mentioned this pull request Jul 9, 2025

ACP: Add a oneshot channel rust-lang/libs-team#610

Closed

Use futex-based synchronization on Apple platforms #122408

Are you sure you want to change the base?

Use futex-based synchronization on Apple platforms #122408

Conversation

joboet commented Mar 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Mar 12, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

madsmtm Sep 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

madsmtm Sep 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bors commented Mar 13, 2024

Uh oh!

madsmtm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

the8472 commented Oct 23, 2024

Uh oh!

rustbot commented Oct 23, 2024

Uh oh!

the8472 commented Oct 23, 2024

Uh oh!

madsmtm commented Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joboet commented Feb 3, 2025

Uh oh!

joboet commented Feb 3, 2025

Uh oh!

m-ou-se commented Feb 6, 2025

Uh oh!

bors commented Feb 10, 2025

Uh oh!

BlackHoleFox commented Feb 10, 2025

Uh oh!

RalfJung commented Aug 26, 2025

Uh oh!

madsmtm commented Aug 26, 2025

Uh oh!

Uh oh!

joboet commented Mar 12, 2024 •

edited

Loading

madsmtm Sep 26, 2024 •

edited

Loading

madsmtm Sep 26, 2024 •

edited

Loading

madsmtm commented Oct 23, 2024 •

edited

Loading