android: make replacement-method dispatch lock-free (arm64 deadlock fix)

[Kolektori]

MR: android — make replacement-method dispatch lock-free

Title: android: make replacement-method dispatch lock-free (arm64 deadlock fix)

Branch: fix/android-arm64-replacement-dispatch-deadlock
Commit: dde975d (+ repro/ files)

---

Summary

Installing .implementation hooks can wedge the target process on
arm64: threads pile up blocked on a global GMutex that the
replacement-method dispatch takes on ART's hottest dispatch paths, and a
GC SuspendAll or thread flip starting behind the lock holder freezes
every mutator thread. The process never recovers. x86_64 is unaffected;
disabling the ART JIT reduces but does not eliminate the deadlock.

Root cause

makeArtController()'s dispatch trampolines took a global GMutex on
code paths that run on arbitrary ART threads in Runnable state:

• instrumentArtQuickEntrypoints() patches the prologues of
  art_quick_generic_jni_trampoline, art_quick_to_interpreter_bridge,
  and art_quick_resolution_trampoline, redirecting them to a trampoline
  that calls find_replacement_method_from_quick_code on every
  native-method dispatch, hooked or not. That function took the GMutex
  via get_replacement_method.
• The same lock was taken from the JS thread (hook install/revert) and
  from GC / thread-flip callbacks (synchronize_replacement_methods,
  on_leave_gc_concurrent_copying_copying_phase, run-flip hooks).

A thread blocked on that futex while in ART Runnable state does not
respond to suspend requests. When a SuspendAll or thread flip starts
behind a lock holder (or a thread queued behind one), suspension never
completes and every mutator parks at its next suspend point: a
process-wide freeze. arm64 timing makes the race fire; x86_64 mostly
does not.

Fix

Hook install/revert always runs on the JS thread (single writer), so the
methods / replacements tables can be published as an immutable
singly-linked list:

• Writers prepend a fresh {original, replacement} entry and republish
  the head with a plain aligned pointer store. Each entry is fully
  initialized before it is linked in, and aligned pointer loads/stores
  are atomic and observed in order on arm64/x86-64, so a reader that
  sees the new head also sees the whole chain.
• Readers (dispatch trampolines, GC / thread-flip callbacks) walk the
  list lock-free. Hook counts are typically small (tens of methods), so
  the linear scan is negligible on the dispatch path.
• Deleting a mapping (hook revert, rare) republishes a fresh list
  without the entry; retired lists are deliberately leaked, as there is
  no safe reclamation point without suspending all ART threads.

This removes the only blocking locks from the dispatch trampolines and
from the GC / thread-flip callbacks. The GMutex, both GHashTables,
and their JS-side allocations are gone entirely (net −4 lines).

Reproduction

A standalone repro ships in repro/ (script + README + optional Java
worker source): it hooks frequently-called native methods, drives GC /
flip activity, and prints a heartbeat with a Java round-trip probe.

npm install frida-java-bridge frida-compile   # released npm = unfixed
npx frida-compile repro/repro-deadlock-entry.js -o repro-bundled.js
frida -U -f com.android.settings -l repro-bundled.js

See repro/README.md for the full recipe (frida-server setup, optional
JIT-off, expected results and how to confirm the signature).

Expected, unfixed: the process wedges within minutes — heartbeats
stop and the app ANRs. The trace (kill -3 → /data/anr/anr_*, or
debuggerd -b <pid>) shows several threads blocked in
pthread_mutex_lock at the same PC in the dispatch trampoline, entered
from art_quick_generic_jni_trampoline:

#02 NonPI::MutexLockWithTimeout(pthread_mutex_internal_t*, ...)
#03 pthread_mutex_lock
#04 <dispatch trampoline code>     <- same PC across threads
#08 art_quick_generic_jni_trampoline

Expected, fixed: no dispatch-mutex cluster; the app's threads stay
responsive; the run completes without a wedge.

Validated on arm64 Android 13/14 emulators: the unfixed bridge wedges
with the same-PC mutex cluster under art_quick_generic_jni_trampoline;
with the fix, no thread is ever observed blocked in the dispatch path.

Notes

• The ANR-trace signature is the ground truth for this bug — the
  same-PC pthread_mutex_lock cluster under the generic JNI trampoline.
• Secondary observation (not addressed here): patching the quick
  entrypoints with Memory.patchCode without suspending ART threads is
  racy on arm64 (only single-instruction B/BL/NOP/BRK patching is safe
  against concurrent execution). Under heavy GC churn with the JIT
  enabled, the workload can trip that race and crash instead of
  deadlocking — the JIT-off path in the repro avoids this so the
  deadlock signature is what you observe. A follow-up could suspend ART
  threads during hook install or adopt an ARM-conformant patch protocol.