feat(gdb): Multithreading support by fogti · Pull Request #1298 · hermit-os/uhyve

fogti · 2026-03-10T14:37:53Z

This is an experiment.

Supersedes #1196.
Fixes #1088.

codecov · 2026-03-10T14:42:45Z

Codecov Report

❌ Patch coverage is 85.39604% with 59 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.58%. Comparing base (7dea91c) to head (c124f38).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
src/linux/mod.rs	64.91%	20 Missing ⚠️
src/linux/gdb/resume.rs	86.13%	14 Missing ⚠️
src/lib.rs	0.00%	12 Missing ⚠️
src/linux/gdb/mod.rs	92.20%	12 Missing ⚠️
src/linux/gdb/breakpoints.rs	98.41%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1298      +/-   ##
==========================================
+ Coverage   75.79%   76.58%   +0.78%     
==========================================
  Files          27       29       +2     
  Lines        4033     4266     +233     
==========================================
+ Hits         3057     3267     +210     
- Misses        976      999      +23

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fogti · 2026-03-10T23:29:02Z

The commits before the multithreading [WIP] one do afaik work as-is (and are mergable, maybe I'll port them to a separate branch).

The rest is still a bit WIP while I and @n0toose iron out the remaining bugs.

fogti · 2026-03-11T01:19:01Z

Besides the mentioned issue (which is a fixable race-condition), this should now mostly work.

fogti · 2026-03-11T01:28:40Z

And the failure that I think stems from the issue mentioned above is:

/tmp/.tmphJS6yJ/commands:5: Error in sourced command file:
Remote connection closed
test gdb ... FAILED

failures:

---- gdb stdout ----
Building test kernel: gdb
Waiting for a local GDB connection on port 1234...
Debugger connected from 127.0.0.1:41290

thread '<unnamed>' panicked at src/linux/mod.rs:123:26:
GDB incoming_data error: GdbStubError { kind: PacketParse(MalformedCommand) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'gdb' panicked at tests/gdb.rs:96:5:
assertion failed: status.success()


failures:
    gdb

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.96s

error: test failed, to rerun pass `--test gdb`

fogti · 2026-03-11T01:32:08Z

Successful gdb invocation now look like this:

(gdb) break multi_thread::main
Breakpoint 1 at 0x26b787: file src/bin/multi-thread.rs, line 14.
(gdb) target remote :6677
Remote debugging using :6677
0x000000001f4e7240 in _start ()
(gdb) c
Continuing.

Breakpoint 1, multi_thread::main () at src/bin/multi-thread.rs:14
14		println!("Multi threading test");
(gdb) step 100
core::alloc::layout::Layout::pad_to_align (self=0x400002519e98)
    at /home/fogti/.rustup/toolchains/nightly-2025-06-01-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/alloc/layout.rs:339
339	       let new_size = self.size_rounded_up_to_custom_align(self.align);
(gdb) c
Continuing.
[Inferior 1 (process 1) exited normally]
(gdb) exit

fogti · 2026-03-11T09:42:21Z

There are still cases where this runs into

/tmp/.tmpLEcOLe/commands:5: Error in sourced command file:
Remote connection closed
test gdb ... FAILED

failures:

---- gdb stdout ----
Building test kernel: gdb
Waiting for a local GDB connection on port 1234...
Debugger connected from 127.0.0.1:41622

thread '<unnamed>' (21473) panicked at src/linux/mod.rs:121:26:
GDB incoming_data error: GdbStubError { kind: PacketParse(MalformedCommand) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'gdb' (21446) panicked at tests/gdb.rs:96:5:
assertion failed: status.success()


failures:
    gdb

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.30s

error: test failed, to rerun pass `-p uhyve --test gdb`

but idk what causes those.

fogti · 2026-03-11T09:54:31Z

uhyve-gdb-malformed-01.pdf
is an example TCP stream of a session that ended with the above malformed packet error.

fogti · 2026-03-11T10:05:05Z

uhyve-gdb-not-malformed-01.pdf
And an example of a successful run.

fogti · 2026-03-11T10:11:28Z

ohno

bad:  $vCont;s:p1.-358a3940#bf
good: $vCont;s:p1.527f66c0#c4

We might have to truncate thread ids to 31 bits instead of 32 bits...

fogti · 2026-03-11T10:30:47Z

Looks good now.

n0toose · 2026-03-11T10:31:40Z

We might have to truncate thread ids to 31 bits instead of 32 bits...

Yeah, I just noticed that when looking at this:

jounathaen · 2026-03-11T16:49:44Z

 }

-impl Target for GdbUhyve {
+impl Freewheel {


I'd prefer to move the Freewheel impls up to be directly after the struct declaration.

fogti · 2026-03-16T21:10:38Z

Unfortunately, a force-push was necessary to get rid of the merge-conflict.

fogti · 2026-03-16T21:22:02Z

cc @n0toose I had to squash fogti#1, I hope that's okay.

also, don't forget to provide me your latest results with regarding the GDB architecture-independence stuff.

jounathaen · 2026-03-17T10:17:27Z

I've created a follow-up issue so we can merge this now: #1314

@fogti @n0toose should we keep the 35 commits, or is squashing the way to go?

n0toose · 2026-03-17T10:20:40Z

I think separating concerns would work better and would suggest that not expanding the original scope of the PR would work best.

I currently have rather limited capacities; I can't predictably tell when I'd have the time to combine some of the changes with the tree that was worked on in parallel. Planning accordingly would be my advice.

n0toose · 2026-03-17T10:21:07Z

squashing the way to go

I'd be for squashing, e.g. for the typo fixes.

fogti · 2026-03-17T17:18:51Z

Should I squash this PR and put the squashed version on this branch, or should I leave this as-is and let @jounathaen squash-merge this?

n0toose · 2026-03-18T07:50:54Z

Former, I believe.

jounathaen · 2026-03-18T08:44:42Z

Please squash it.

fogti · 2026-03-18T08:49:32Z

Okay, before I do that, I first try to merge this into the arch-indep branch version, in order to not have to untangle that later.

Co-authored-by: Panagiotis "Ivory" Vasilopoulos <git@n0toose.net>

fogti · 2026-03-18T09:23:04Z

Okay, squashed all the chore/nit commits into one, and added reference URLs for the gdbstub Tid oddity.

This is a combination of 30 commits. nit(gdb): remove single gdb restriction feat(gdb): set affinity in threads, fetch vcpu id from object nit(gdb): Improve cpu_affinity handling nit(gdb): have swap use AcqRel ordering fix(gdb): GDB truncates thread-ids supplied by us, so truncate them before supplying them to GDB to ensure consistency ``` [TRACE uhyvelib::linux::gdb] tid2vcpu = {140098616854208: 0} [TRACE uhyvelib::linux] Active thread: 140098616854208 [TRACE uhyvelib::linux::gdb] tid_to_vcpuw(1078625984) thread 'main' (4706) panicked at src/linux/gdb/mod.rs:237:37: no entry found for key note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ``` ``` >>> 140098616854208 & 1078625984 1078625984 >>> 140098616854208 ^ 1078625984 140097538228224 >>> x = 140098616854208 ^ 1078625984 >>> f"{x:#x}" '0x7f6b00000000' ``` fix(gdb): Fix most cases of GDB not being able to interrupt the target These were actually multiple issues in a trenchcoat: - set_scheduler_lock is called very late (often just before resume) - for cleanliness, we now always check if we're stopped (per vcpu), even on first iteration - wait in the vcpu threads for the gdb connection to be fully initialized, to ensure that the breakpoints are set correctly (`full_init_event`) It appears that this leads to a breakpoint getting hit in _start, but at least it otherwise appears to work perfectly almost always. Remaining failures are probably due to the non-intended treatment of `full_init_event` (it should've been paired with an `AtomicBool` to ensure spurious wakeups don't lead to protocol errors). fix(gdb): Truncate TIDs to be provided to GDB to 31 bits See also: - daniel5151/gdbstub#193 - https://sourceware.org/bugzilla/show_bug.cgi?id=33979 nit(gdb): Rename 'struct Freewheel' to 'struct GdbVcpuManager' nit(gdb): Rename 'fn backend_report_invalid_value' to 'fn backend_invalid_value' nit(gdb): Rename Tid resolvers nit(gdb): Fix VcpuWrapper::tid comment nit(gdb): Rename 'ResumeMode::Freewheel' to 'ResumeMode::FreeWheeling' nit(gdb/resume): renamse '.free_wheel' to '.r#continue' nit(gdb): Rename '.finished_initializing' to '.set_finished_initializing' chore(gdb): Mark GdbVcpuManager getters as non-pub Co-authored-by: Panagiotis "Ivory" Vasilopoulos <git@n0toose.net>

fogti · 2026-03-18T09:29:47Z

(sorry, had to fix the attribution in the second multithreading commit) (this is now ready to merge, afaik)

jounathaen · 2026-03-18T10:44:30Z

Thank you both!

fogti force-pushed the gdb-stub-inline branch 3 times, most recently from 4659754 to c3cfe88 Compare March 10, 2026 23:21

fogti changed the title ~~chore(gdb): inline run_blocking to make further refactoring easier~~ feat(gdb): Multithreading support Mar 11, 2026

fogti commented Mar 11, 2026

View reviewed changes

Comment thread src/linux/gdb/mod.rs Outdated

fogti mentioned this pull request Mar 11, 2026

Draft multicore gdb #1196

Closed

n0toose requested review from jounathaen and mkroening March 11, 2026 08:25