Skip to content

WHATS_WORKING.md

Latest commit

 

History

History
630 lines (511 loc) · 36.8 KB

WHATS_WORKING.md

File metadata and controls

630 lines (511 loc) · 36.8 KB
Raw

What Works Today in Hermit

This document is a practical, hands-on snapshot of what Hermit can do right now. The baseline matrix below was generated by actually running each program under hermit run --strict --verify on main at commit db09337 (backend = ptrace, the default). Sections 3p-3q add the overnight expansion evidence from batches 3-41 and the real-application matrix. It is meant to be copy-pasteable so you can reproduce the results yourself.

TL;DR: On the default ptrace backend, Hermit runs a broad range of real programs bit-for-bit deterministically under --strict --verify — all of coreutils, crypto/hashing, compression, SQLite, floating-point math, filesystem tools, process trees, and multithreaded C/OpenMP programs whose threads do not read wall-clock time. 75 distinct program invocations were exercised in the baseline pass. The overnight expansion produced 134 clean L2 results as written, or 135 valid L2 results after replacing one malformed socket command with its corrected AF_UNIX form. A second expansion (batches 30-41) added 56 more clean L2 passes across networking/sockets, signals, time/clock, environment/argv, mmap/memory, epoll/poll, larger CPython, C data-structure apps, Perl, C++, and compiled Rust, plus five real userland apps (bc, awk, sort, sed, sqlite3) verified at L2 (§3q). These batch rows overlap the baseline and each other, so they are not added to 75 as a distinct-program total. Recurring engine gaps include multithreaded wall-clock reads, heavy compiler process trees, and blocking FIFO rendezvous. The clock gap is strongly load-sensitive (see the box in §3); NSS and /proc mismatches additionally expose non-hermetic host state. New this session: Hermit wraps a full QEMU VMM and boots Linux to userspace with byte-identical serial output across two runs under relaxed flags (§3r), and a consolidated 9-language L2 matrix (C, C++, Rust, Go, Perl, Lua, Node.js, Java, gawk) verifies clean (§3s).

⚠️ Run conditions matter. This matrix was captured on a heavily loaded host (load average ≈ 33, dozens of concurrent agents). The multithreaded-clock divergence is timing/load-sensitive: programs that read the clock from worker threads (notably CPython: python3, and anything embedding it) pass on a lightly-loaded host but fail --verify frequently under this load. Where a result is load-sensitive it is labeled FLAKY with the observed pass rate, not a flat PASS. This is the honest overnight picture.

1. Building Hermit

cd ~/work/dev-hermit/hermit
cargo build -p hermit
# binary lands at: ./target/debug/hermit

A release build (faster guests) is cargo build -p hermit --release → ./target/release/hermit. All results in this doc are from the debug binary.

For convenience the rest of this doc assumes:

HERMIT=$(pwd)/target/debug/hermit

2. How to test a program deterministically

The core value proposition is deterministic execution. Run any program under strict mode and ask Hermit to re-run it and diff the two executions:

$HERMIT run --strict --verify -- <program> [args...]
  • --strict — full deterministic mode (virtual time, virtualized PIDs/ports, deterministic scheduling). It is currently the default; the flag is kept for clarity.
  • --verify — run the program twice and confirm the two runs are bit-for-bit identical. On success you get: :: Success: deterministic. Determinism verified. On failure: :: Failure: nondeterministic.

Two useful notes about the sandbox:

  • Hermit isolates the guest /tmp and virtualizes the filesystem view, so process substitution (<(...) → /dev/fd/N) and host-only paths are not visible to the guest.
  • Hermit virtualizes ephemeral ports: bind(("127.0.0.1", 0)) + getsockname() returns a deterministic 32768 every run.

3. Verified pass/fail matrix (--strict --verify, backend = ptrace)

Legend: PASS = Determinism verified observed. FLAKY (n/m) = passed n of m repeat runs (load-sensitive). FAIL = reproducible --verify mismatch. native rc=1 = the program exits non-zero on the host too (bad args / environment), not a Hermit determinism issue. TIMEOUT = did not finish within 90 s under this load.

Single-threaded compute determinism is rock-solid. As a control, sha256sum was run 10× back-to-back: 10/10 PASS. Baseline failures were multithreaded wall-clock reads or NSS/socket interactions. The expansion also characterizes heavy compiler process trees and a blocking FIFO-open rendezvous; neither is a single-threaded compute workload.

3a. Coreutils / text processing — 21/21 deterministic

App Result App Result
true PASS tac PASS
echo PASS rev PASS
cat PASS paste PASS
wc PASS join PASS
head PASS fold PASS
tail PASS od PASS
sort PASS factor PASS
uniq PASS nl PASS
sed PASS cut PASS
awk PASS tr PASS
seq PASS comm native rc=1 (needs sorted input)

3b. Crypto / hashing — 7/7 PASS

App Command Result
md5sum md5sum FILE PASS
sha1sum sha1sum FILE PASS
sha256sum sha256sum FILE PASS
base64 base64 FILE PASS
openssl dgst openssl dgst -sha256 FILE PASS
openssl rand openssl rand -hex 16 PASS — RNG virtualized → deterministic
openssl enc openssl enc -aes-256-cbc -pass pass:x -pbkdf2 -in FILE PASS — random salt virtualized

3c. Compression — 5/5 PASS

App Result
gzip -c PASS
bzip2 -c PASS
xz -c PASS
zstd -c PASS
tar --version PASS

3d. Database — 2/2 PASS

Command Result
sqlite3 :memory: 'SELECT 1+1;' PASS
sqlite3 :memory: 'CREATE TABLE…; INSERT…; SELECT sum(x)…' PASS

3e. Floating-point math — 4/4 PASS

App Command Result
C libm 1000-iter sqrt()*sin() sum, %.10f PASS
bc bc -l PASS
expr expr 6 \* 7 PASS
perl FP perl -e 'printf("%.6f", atan2(1,1)*4)' PASS

3f. Filesystem — 6/6 deterministic

App Result
stat PASS
readlink -f PASS
realpath PASS
dd PASS
cmp (identical files) PASS
du -b PASS
diff (differing files) native rc=1 (files differ; deterministic, non-zero exit)

3g. Environment / locale — 5/6 (one NSS flake)

App Result
printenv PASS
env -i (clean env) PASS
uname -a PASS
nproc PASS
locale PASS
id FLAKY (9/10 PASS) — see NSS note §3h

3h. NSS consumers (getpwuid/getgrgid) — deterministic most of the time

id, ls -la, whoami, getent, and tar's owner-name listing all call glibc NSS (getpwuid/getgrgid). Detcore has no NSS handlers; these resolve via /etc/passwd file reads or a connect() to the host nscd/name-service socket, and that socket path is host-timing-dependent.

App Result
id FLAKY (9/10 PASS) — ~10% --verify divergence
ls -la PASS (6/6 this sample; known to flake under load)
whoami PASS
getent passwd root PASS
tar -tvf native rc=1 (empty archive)

3i. Timers / signals — 3/3 PASS

Test Result
sleep 0.05 PASS
date -u +%Y (virtual time) PASS
C signal(SIGUSR1) + raise() handler PASS

3j. Multithreaded — threads that don't read the clock PASS; threads that do FAIL

Test Result
C pthreads ×4 (gcc -pthread) PASS
OpenMP parallel-for reduction ×N (gcc -fopenmp) PASS
Python threading ×4 (start/join) FAIL — worker-thread clock_gettime sub-second divergence

Thread scheduling determinism — the core value prop — holds: C pthreads and OpenMP spawn real threads with racy ordering and come out bit-identical. The multithreaded failures are exclusively wall-clock reads from worker threads, not scheduling.

3k. Process trees — 3/3 PASS

Test Result
bash -c 'for i in 1 2 3; do echo $i; done' PASS
nested subshell tree (echo a; (echo b; echo c)) PASS
pipeline `seq 1 100 grep 5

3l. Language runtimes — mixed (CPython is the load-sensitive outlier)

Runtime Command Result
perl perl -e 'print 6*7' PASS
lua lua -e 'print(6*7)' PASS
node node -e 'console.log(6*7)' PASS
java java -version PASS (OpenJDK 1.8.0_492 Temurin, 5/5 runs) — requires PR #223 (saturating_add fix for LogicalTime overflow, #219)
gawk gawk 'BEGIN{print 6*7}' PASS
python3 (pure compute) python3 -c 'print(sum(range(100)))' FLAKY (2/10 PASS under load) — worker-thread clock_gettime(CLOCK_MONOTONIC_COARSE) sub-second divergence (dtid 3). Passes on a lightly-loaded host.
python3 threading see §3j FAIL — same clock divergence
php php -r 'echo 6*7;' TIMEOUT (>120 s under load; heavy interpreter startup)
ruby ruby -e 'puts 6*7' native broken — host RubyGems load error (fails outside Hermit too)

3m. Network — loopback

Test Result
curl --version PASS
Python socket.bind(("127.0.0.1",0)) + getsockname() FLAKY (0/8 under load) — the port virtualizes to 32768, but CPython's worker-thread clock_gettime diverges (same root cause as python3); passes on a lightly-loaded host

Loopback only. Hermit does not make external network traffic deterministic (by design).

3n. Servers — 1/1 PASS

Test Result
Redis: redis-server + redis-cli SET foo bar / GET foo / SHUTDOWN NOSAVE (wrapped in one shell script) PASS — full server workflow is bit-for-bit deterministic, stable across 2 runs

Because hermit run takes a single program, multi-step workflows (like the Redis one) are wrapped in a shell script and run as $HERMIT run --strict --verify -- /bin/bash script.sh.

3o. Deterministic builds (the core use case)

Two independent --strict compiles of the same source produce byte-identical binaries (same SHA-256), i.e. reproducible builds:

Compiler Test Native Under Hermit (2 independent --strict runs)
gcc 11 hello.c → binary identical IDENTICAL (sha256 508f2b57…, both exit 0, runs)
clang hello.c → binary — IDENTICAL (sha256 4e298afe…)
gcc 11 program embedding __DATE__/__TIME__ DIFFERENT (bakes wall-clock into the binary) IDENTICAL (sha256 55f52fc2…)

The __DATE__/__TIME__ case is the decisive demonstration: a program that compiles to different bytes natively (the compiler embeds the wall-clock compile time) compiles to identical bytes under Hermit, because the virtualized clock resolves __TIME__ to the same fixed value every run.

printf '#include <stdio.h>\nint main(){printf("hi\\n");return 0;}\n' > hello.c
$HERMIT run --strict -- /usr/bin/gcc -o hello1 hello.c
$HERMIT run --strict -- /usr/bin/gcc -o hello2 hello.c
cmp hello1 hello2 && echo IDENTICAL   # -> IDENTICAL

Note: this is the build artifact being reproducible across independent runs. gcc still fails --strict --verify (that checks internal syscall-trace determinism across a multi-process cc1/as/ld pipeline, see §6) — both facts are true and not contradictory: the emitted binary is deterministic even though the internal syscall interleaving is not.

3p. Overnight expansion batches 3-29 — 135 valid L2 checks

The following results come from the closed batch task notes, not from a single fresh run at the current HEAD. The notes explicitly bind several batches to c88bc0f and the later batches to 21a6813; other batches identify main and target/debug/hermit but omit an exact SHA. Every row used the default ptrace backend, default log level, and no determinism relaxations. Batches 19-20 and 27-28 used --tmp=/tmp only to expose their host-compiled fixture binaries to the guest. PASS L2 means hermit run --strict --verify completed with bitwise-identical repeat output.

The 27 numbered batches contain 141 as-written checks, and the separate interpreter expansion contains 6. Of those 147 checks, 134 passed cleanly as written. Batch 13 contained a malformed Python socket command; replacing it with the intended AF_UNIX abstract-socket bind passed at L2, yielding the 135 valid passing checks catalogued below. Counts are invocation rows, not a deduplicated program count.

Category Batch Passing commands / workloads L2 passes
Shell and text processing 3 echo|sort; sort|uniq; echo|wc; head; sed; awk; bc; date +%s; env|head 9
Language interpreters interpreter expansion CPython JSON, numeric loop, and PID/PPID probes; Perl hello; Node hello 5
Compiled C and primitives 4 malloc/file I/O; fork+pipe+waitpid; four pthreads with mutex; SIGUSR1 handler 4
System utilities 5 find|head; du; readlink; hostname; whoami; tr; tee; basename; dirname 9
Crypto and encoding 6 openssl dgst; sha256sum; md5sum; base64|head; xxd|head 5
Text pipelines 7 passwd sort|head; usernames awk|sort; grep -c; echo|rev; seq|paste|bc 5
Multi-process 8 background children+wait; nested background subshells; Bash coprocess; Python fork() 4
File I/O 9 create/read/unlink; dd+wc; mktemp; touch+ls; cp+diff 5
Compression and archives 10 gzip round-trip; bzip2 round-trip; xz round-trip; zip create/list 4
Process and signals 11 kill -l; trap+SIGUSR1; ulimit -a; getconf PAGE_SIZE 4
Math and numeric 12 seq|factor; bc -l pi; Bash hex conversion; numfmt; expr 5
Networking and identity 13 getent hosts; getent services; Python gethostname; corrected AF_UNIX abstract bind; curl --version 5
Environment and locale 14 env|sort|head; printenv PATH; locale; Bash set; export/unset 5
Build-tool frontends 15 gcc --version; make --version; cargo --version 3
Larger programs 16 rustc --version; Python version/platform; Python environment JSON; sequential wc; find|wc 5
Native Rust programs 17 hello; three spawned/joined threads; create/read/remove file 3
Networking and sockets 18 Python socket create/close; gethostname; resolv.conf|head 3
Process and signals 19 fork+waitpid+virtual PID; SIGUSR1 handler; alarm+pause; pipe+fork; pthread mutex counter 5
Multithreaded programs 20 pthread mutex; producer/consumer condvars; TLS; eight-thread compute; Rust thread reduce 5
System information 21 uname -a; cpuinfo|head; df -h /; uptime 4
Text processing 22 awk; sort; uniq -c; tr; cut in short two-stage pipelines 5
Compression and encoding 23 gzip, bzip2, xz, and zstd round-trips; base64 encoding 5
Math and computation 24 Python constants, sum, seeded random, and permutations; bc -l pi 5
Structured data 25 Python SQLite aggregate; deterministic JSON, CSV, and XML serialization 4
Regex and strings 26 Python regex/string transforms; Perl regex; grep -oE pipeline 5
Concurrency stress 27 16-thread mutex; rwlock; barrier; semaphore producer/consumer 4
Process management 28 virtual identity; fork+wait; four-thread gettid; sysconf; Python identity 5
I/O and filesystem 29 temp-file write/read; stat; sorted readdir; Python tempfile; wc 5
Total 135

The Rust thread probe is a useful scheduling result: Hermit produced a stable thread order that differed from native execution but was byte-identical across both verification runs. The file, crypto, numeric, and pipeline probes also checked expected output, not only matching detlogs.

Non-clean and corrected batch cases

These results are deliberately excluded from the 135-pass table:

Batch Command / workload Observed result Classification
3 id 9/10 passed; one verify mismatch Flaky NSS lookup through the live host nscd socket
interpreter expansion Ruby hello Default invocation fails natively because RubyGems is broken; --disable-gems passes L2 Host runtime failure; corrected control passes
4 gcc hello.c -o ... 2/8 passed; six detlog mismatches Heavy compiler process-tree/InternalIOPolling divergence
5 default-format stat /etc/passwd Intermittent failure, about 1/5 UID/GID name lookup through nscd; numeric format passes 5/5
8 blocking FIFO writer+reader Hangs in plain strict mode and verify FIFO-open rendezvous livelock
10 tar with owner names 7/8 passed NSS/nscd owner-name lookup flake
11 ps aux|head Stdout mismatch with matching detlogs Live %CPU/VSZ/RSS values for the supervisor from /proc
13 Python AF_INET bind with a NUL hostname Fails natively with TypeError Malformed test; corrected AF_UNIX form is counted above
15 Meta git --version; git log -1 Both hang even in one strict run Site-specific telemetry subprocess interaction
18 Python getfqdn() 4/5 passed; one resolver error Flaky host NSS/DNS resolution, not a repeat-output mismatch
18 ss -tlnp 0/5; detlog mismatch Reads changing live netlink socket state and /proc/<pid>/fd
21 meminfo|head 0/3; stdout mismatch with identical detlogs Dynamic host memory counters are passed through, not snapshotted

These classifications matter: a host-state mismatch or invalid command is not a clean L2 pass, but it is also not evidence of a missing syscall handler. The reproducible product gaps remain the compiler process-tree divergence and FIFO rendezvous livelock; NSS and /proc rows require additional isolation or virtualization to become deterministic.

3q. Second expansion batches 30-41 — 56 clean L2 passes + real-app matrix

A second overnight expansion (batches 30-41) probed twelve program families under hermit run --strict --verify (backend ptrace, default log level, no determinism relaxations). Results come from the closed batch task notes; batches bind to main at 163333c / 859970c / 21a6813 (each note records its SHA). Every row was confirmed both deterministic across the two --verify sub-runs and byte-identical to native stdout under a separate --strict run. PASS L2 = :: Success: deterministic. Determinism verified.

Setup note (recurring across these batches): current main refuses a guest program located under host /tmp (--strict errors "Program /tmp/… is under host /tmp … Pass --tmp=/tmp"), because Hermit isolates the guest /tmp. Batches that "compile to /tmp" therefore relocated the binary outside /tmp (e.g. ~/…) or passed --tmp=/tmp. This is the isolation guard, not a determinism failure.

Batch Family Passes Representative workloads
30 Networking / sockets 5/5 AF_UNIX socketpair (STREAM+DGRAM), python3.9 socketpair, pipe2(O_CLOEXEC), eventfd
31 Signals 4/4 signal+raise, alarm(1)+pause, sigaction SA_SIGINFO, sigprocmask block/pending
32 Time / clock 5/5 clock_gettime MONOTONIC+REALTIME, gettimeofday, clock_getres, python3.9 time.*
33 Environment / argv 5/5 argc/argv, getenv, python os.environ, $PATH split pipe, uname() (canonicalized)
34 mmap / memory 5/5 anon mmap, mprotect→SIGSEGV catch, mremap, madvise(DONTNEED), brk/sbrk
35 epoll / poll 5/5 epoll_wait, poll, select (2 pipes), timerfd, signalfd
36 Larger CPython 3.9 5/5 hashlib.sha256, itertools.permutations, collections.Counter, functools.reduce, struct.pack
37 Real C apps (mini) 5/5 malloc/free stress, linked list, BST, open-addressing hash table, matrix multiply
38 Perl (Ruby N/A) 4/4 POSIX strftime, hash sort-keys, map squares, /etc/passwd line count
39 Multithreaded data structures 5/5 condvar prod/cons queue, pthread_rwlock, TLS, raw-futex mutex ×40000, thread pool
40 C++ (g++ -std=c++17) 5/5 std::vector+sort, std::map, std::thread+atomic ×400000, std::regex, std::chrono::steady_clock
41 Compiled Rust 3/3 integer compute, 4× std::thread join, HashMap (seed virtualized)
Total 56

Real userland applications (hermit run --strict --verify -- bash -c '<cmd>') — 5/5 PASS L2:

App Command Result Output
bc bc -l pi via 4*a(1), scale=20 PASS L2 3.14159265358979323844
awk echo '1 2 3 4 5' | awk field sum PASS L2 15
sort printf '3\n1\n4\n1\n5\n' | sort -n PASS L2 1 1 3 4 5
sed echo 'hello world' | sed 's/world/hermit/' PASS L2 hello hermit
sqlite3 CSV CREATE/INSERT 42/SELECT via pipe PASS L2 42

Notable results and honest caveats:

  • Virtual time makes clock-reading programs verify. C std::chrono::steady_clock diff is byte-identical across two independent --strict runs (diff_ns=1010030) though it varies natively; batch 32 pins all clocks to the virtual epoch 1640995199 (2021-12-31 23:59:59 UTC).
  • Synchronous fault + signal paths are deterministic: mprotect(PROT_READ)→SIGSEGV caught via sigsetjmp/siglongjmp (batch 34) and SA_SIGINFO delivery (batch 31) both verify at L2.
  • Raw-futex and lock-heavy multithreading verify (batch 39: 3-state Drepper futex mutex ×40000, rwlock, condvars) — deterministic scheduling, no livelock; single-process threads are the class Hermit determinizes reliably.
  • Batch 31 #5 (python raising an unhandled SIGUSR1) terminates deterministically (exit 138 = 128+10, identical 3/3) but is not --verify-able: the harness treats a signal-terminated (non-zero) first run as an error before the second pass. A --verify limitation for self-terminating guests, not a determinism gap.
  • Batch 38 Ruby is N/A — /usr/bin/ruby is broken natively on this host (RubyGems RbConfig load error, same failure outside Hermit), so no determinism result is possible.
  • The real-app pipelines pass under run --verify even though the same pipelines deadlock under hermit record (record-only pipe issue; see §5).

3r. Nested VMM: QEMU Linux boot — deterministic under relaxed Hermit

Hermit can wrap a full hardware emulator: hermit run launches qemu-system-x86_64 (TCG, software emulation) which boots a real Linux kernel to userspace. Measured 2026-07-23 on main, target/release/hermit, host kernel /boot/vmlinuz 6.17.13 + a busybox static initramfs (rdinit=/init).

Mode Command flags Result
Relaxed (working) --no-sequentialize-threads --preemption-timeout disabled --no-virtualize-cpuid + QEMU -accel tcg,thread=single -smp 1 -icount shift=0,sleep=off BOOTS TO USERSPACE — kernel comes up (e820/ACPI/smpboot CPU0/clocksource tsc/btrfs/ima) → Run /init → busybox shell; exit 0 with an auto-poweroff init
Relaxed, repeated ×2 same, hermit_autotest on cmdline (auto-poweroff) BYTE-IDENTICAL — two independent boots produced identical 22908-byte serial logs, sha256 564e1ba4… (kernel printk timestamps included). Determinism confirmed by direct 2-run comparison
--strict / --strict --verify strict re-enables sequentialize-threads + PMU-preemption single-stepping + cpuid virtualization DOES NOT COMPLETE — QEMU launches but the guest kernel emits zero serial output within 240s (no [0.000000] Linux version); timeout SIGKILL. Not a crash/syscall gap — precise-preemption single-stepping is catastrophically slow for a CPU-bound emulator

Notes:

  • Determinism here is established by a manual two-run byte comparison, not by --strict --verify (which cannot complete in-window). The relaxations are documented requirements, not conveniences — see docs/QEMU_BOOT.md: --no-sequentialize-threads (QEMU needs concurrent host threads), --preemption-timeout disabled (no PMU single-step), --no-virtualize-cpuid (CPUID faulting on this host), -icount shift=0,sleep=off (single instruction-derived guest clock; the alternative is Hermit-side --no-virtualize-time --no-virtualize-metadata).
  • Corroborated by the preserved experiment experiments/qemu-boot-debug/results.csv, row virtual_minimal_fixed_icount → complete_boot, exit 0, coherent 1000.031MHz clock.
  • Under strict, Hermit prints an explicit VMM warning (mutually-inconsistent RDTSC vs virtualized clock_gettime can corrupt guest clock calibration).

3s. Language-runtime summary — 9 languages verify at L2

Consolidated from §3l and batches 36/37/38/40/41, plus a Go check added 2026-07-23. PASS L2 = hermit run --strict --verify reports :: Success: deterministic. Determinism verified. Sources/binaries kept outside the Hermit-isolated /tmp.

Language Witness command Result
C gcc hello.c build + run; batch 37 mini-apps PASS L2
C++ batch 40: g++ -std=c++17 vector/map/std::thread+atomic ×400000/regex/chrono PASS L2 (5/5)
Rust batch 41 + rustc -O integer-sum binary PASS L2 (3/3 this session)
Go go build integer-sum binary (go sum: 4950) PASS L2 (3/3 this session) — Go's multithreaded runtime verifies for this compute workload
Perl perl -e 'print 6*7'; batch 38 (strftime/hash/map/line-count) PASS L2
Lua lua -e 'print(6*7)' PASS L2
Node.js node -e 'console.log(6*7)' PASS L2
Java java -version (OpenJDK 1.8.0_492 Temurin, 5/5) — requires PR #223 (saturating_add LogicalTime overflow fix) PASS L2
gawk gawk 'BEGIN{print 6*7}' PASS L2
Python 3 python3 -c 'print(sum(range(100)))' CONDITIONAL — PASS on a lightly-loaded host; flaky under load (multi-thread clock_gettime sub-second divergence, §6.1)
Ruby ruby -e 'puts 6*7' N/A — host RubyGems broken (fails outside Hermit too), not a determinism result
PHP php -r 'echo 6*7;' N/A — HHVM JIT, too slow to finish twice under load (timeout), not a determinism result

The 9 solidly-verifying languages are C, C++, Rust, Go, Perl, Lua, Node.js, Java (with PR #223), and gawk. Python verifies only on an idle host; Ruby and PHP are excluded for host/runtime reasons that are not Hermit determinism failures.

4. Backend status

Backend Flag Status
ptrace --backend ptrace (default) Working. Every result in this doc uses it.
DBI (DynamoRIO) --backend dbi SDK-gated. Now ungated when a DynamoRIO SDK is present (PR #213). With no SDK it fail-closes: backend 'dbi' is unavailable: the DynamoRIO SDK was not found; set DYNAMORIO_HOME or DynamoRIO_DIR to a valid SDK. No SDK is installed on this host, so the DBI E2E path is untested here.
KVM --backend kvm Wiring in progress. Fail-closed; no Tool/Guest adapter for executing Linux programs yet. 
$HERMIT run --strict --verify -- /bin/echo hi   # ptrace: works
$HERMIT run --backend dbi -- /bin/echo hi        # needs DYNAMORIO_HOME/DynamoRIO_DIR
$HERMIT run --backend kvm -- /bin/echo hi        # fail-closed

5. Record / Replay status

Record an execution and replay it later; the recording captures the deterministic syscall stream.

$HERMIT record start -- /bin/echo hello           # prints a recording id
$HERMIT replay <ID>                               # replays (can drive a debugger)
$HERMIT record start --verify -- /bin/echo hello  # record + self-check the replay
$HERMIT record list | rm <ID> | clean            # manage recordings

The record/replay integration suite is green:

cargo test -p hermit --test record_replay -- --test-threads=1
# test result: ok. 24 passed; 0 failed; 0 ignored

The requested overnight R/R expansion also passed all 16 workload rows. These are raw workload checks, not a deduplicated program count. The first three groups were recorded on main 21a6813; the stress note identifies a detached main-based slot but omits its SHA.

Workload group Passing record+replay checks Result
Computation recursive C Fibonacci; Python sum; Perl exponentiation; Bash builtin arithmetic 4/4
Python in-memory SQLite; sorted JSON; math.pi; numeric loop 4/4
Sequential multi-process sequential external echoes; fork+wait; fork+exec+wait 4/4
Stress eight-thread atomic; condvar producer/consumer; ten Python threads; 100 sequential forks 4/4, each repeated 3/3
Total 16/16

This confirms the passing R/R envelope for single-process compute, multithreaded guests, and sequential fork/exec. It does not cover concurrent children connected by live pipes. Those workloads still expose replay pipe ordering, descriptor tracking, and stdout-routing gaps; the 16/16 result must not be generalized to arbitrary multi-process pipelines.

Additional R/R expansion (record start --verify, ptrace, main @ 21a6813). A follow-up sweep of syscall families under record/replay sharpened the single-process-passes / pipeline-hangs boundary:

Family Workloads record --verify
Signals (single-process) raise(SIGUSR1)+handler; alarm+pause+SIGALRM; sigaction SA_RESTART PASS — replay byte-clean
I/O & files open/write/read/close temp file; stat("/etc/passwd"); bash -c 'wc -l FILE'; python3.9 os.stat PASS (4/4; python stable 3/3)
Readiness / notify epoll_wait; poll; timerfd blocking read (stable 3/3); eventfd PASS (4/4)
Filter apps, file input bc/awk/sort/sed FILE (single-process) PASS (4/4)
Compiled languages rustc -O and go build integer-sum binaries (single-process) PASS (2/2, added 2026-07-23) — :: Success: replay matched recording. — extends the compute-R/R envelope (C/Python/Perl/Bash) to Rust and Go
Filter apps, shell pipe echo … | bc/awk/sort/sed HANG — record pipeline deadlock (record-deadlock since fixed by PR #230/#235; a replay stdout-doubling gap remains, so pipe R/R is not yet --verify-clean)
fork + cross-process signal fork() + parent write() + kill(child) + waitpid HANG — scheduler deadlock

Two reproducible R/R-record gaps were pinned:

  1. Concurrent shell pipelines deadlock hermit record (2 processes + a live pipe). The identical pipeline is deterministic under plain run --verify, and the same apps record cleanly when fed a file argument instead of a pipe — so the deadlock is record-engine specific, not an app or determinism defect.
  2. A parent write() immediately followed by a cross-process kill() of a child blocked in pause() deadlocks the scheduler (a BlockingExternalIO turn-race). A non-I/O syscall before the kill does not trigger it; single-process signal delivery and in-guest (socketpair/pipe) fork IPC are clean.

Note: unlike --verify (which recomputes virtual time on both runs), record/replay records the real host clock and replays the bytes, so clock-heavy programs replay cleanly on that path.

6. Known limitations & open issues

Real Hermit engine gaps (worth follow-up):

  1. Multithreaded wall-clock reads diverge under --verify. This is the dominant remaining gap and it is load-amplified. When a program reads clock_gettime/gettimeofday from more than one thread (CPython does this internally, so plain python3 is affected — not just python3 threading), the sub-second component of virtual time differs run-to-run (same fixed epoch second, divergent nanoseconds; observed on CLOCK_MONOTONIC_COARSE, dtid 3). Root cause: Detcore's guest clock returns a global sum across all threads' work incorporating PMU-derived RCB counts; the structural bug is clone-time shared-prefix double-counting whose magnitude is RCB-jitter-sensitive (see detcore-model/src/time.rs GlobalTime/update_global_time, detcore/src/lib.rs clone site). Single-threaded time is exact. On a lightly-loaded host these programs pass; under the load this matrix was captured at (avg ≈ 33) they fail most of the time.

  2. NSS lookups are host-socket-dependent (flaky). id/ls -la/tar owner-name listings call glibc getpwuid/getgrgid, which may connect() to the host nscd/name-service socket. Detcore does not virtualize that daemon; the connect/recv completes at a host-timed moment, perturbing the deterministic schedule (~10% --verify divergence for id). A hermetic /etc (files-only nsswitch) or intercepting the nscd connect with ECONNREFUSED to force the deterministic files fallback would fix it.

Changed since the last snapshot:

  • openssl speed no longer crashes. The previous scheduler panic in scheduler/timed_waiters.rs (SIGALRM/setitimer re-arm) appears fixed by PR #214. openssl speed -evp aes-256-cbc -seconds 1 now runs to completion; its --verify result is a benign benchmark-output mismatch (it prints measured ops/sec, which is inherently timing-derived) rather than a SIGSEGV. The syscall DETLOG shows "no substantive differences."

  • PRs landed (2026-07-23). Eleven PRs merged across the two repos this session:

    Repo PR Title
    hermit #211 validate: auto-apply locally-validated PR label on a green run
    hermit #225 DBI M2a: add reverie-dbi dependency to hermit-cli
    hermit #229 KVM M3: prove Detcore drives KvmGuest via run_with_tool
    hermit #230 detcore: classify internal pipes as InternalIOPolling (fix R/R pipe record deadlock)
    hermit #233 Fix KVM backend dispatch and private-flag futex timeout classification
    hermit #234 DBI M2b: route DBI backend through reverie_dbi::DbiRunner
    hermit #235 detcore: record internal-pipe read data on the InternalIOPolling path (R/R replay ordering)
    reverie #23 Document and formalize the Reverie backend contract
    reverie #32 reverie-dbi: Guest stack + tail_inject (M2); simple observation tools
    reverie #39 reverie-kvm: StraceTool over KvmGuest (KVM M2)
    reverie #40 reverie-dbi: park/unpark executor so DBI handlers can suspend (async FFI bridge)

    These advance the DBI (DynamoRIO) and KVM backends behind the still-default ptrace backend (#225/#233/#234 + reverie #23/#32/#39/#40) and the record/replay pipe path (#230/#235); the DBI/KVM E2E paths remain gated/in-progress as described in §4, and pipe R/R is not yet --verify-clean (see §5).

Not Hermit issues (documented so they aren't mis-filed):

  • diff <(...) <(...) — process-substitution /dev/fd/N fds aren't in the isolated fs; regular-file diff is deterministic.
  • comm on unsorted input, diff on differing files, tar -tvf /dev/null — all exit non-zero natively (input/args), not determinism failures.
  • ruby -e — host RubyGems is broken (RubyGems were not loaded); fails outside Hermit too.
  • php -r — completes natively but is too slow to finish twice under this host load (TIMEOUT), not a determinism result.

7. Reproducing the full matrix

cd ~/work/dev-hermit/hermit
cargo build -p hermit
HERMIT=$(pwd)/target/debug/hermit

# Any single program:
$HERMIT run --strict --verify -- /bin/echo hello

# A load-sensitive case (pass on idle host, flaky under heavy load):
$HERMIT run --strict --verify -- python3 -c 'print(sum(range(100)))'

# Record/replay suite:
cargo test -p hermit --test record_replay -- --test-threads=1

Multithreaded / network programs are compiled natively and run under Hermit, so these measure runtime determinism (thread scheduling, syscalls, virtual time). Keep sources and binaries outside the Hermit-isolated /tmp when reproducing. Because the multithreaded-clock gap is load-sensitive, quantify flaky cases with repeat runs (e.g. 10×) rather than a single trial, and record the host load average alongside the result.