feat(opentmk): opentmk framework with first testcase #1210

mayank-microsoft · 2025-04-21T18:48:23Z

No description provided.

feat: opentmk init feat: opentmk init feat: opentmk init feat: opentmk init feat: opentmk init feat: opentmk init feat: init 1 feat: init 2 feat: init 1 feat: opentmk feat: opentmk init 3 feat: opentmk init 4 feat: opentmk init 4

…ll the registers

smalis-msft · 2025-04-22T15:26:54Z

Cargo.toml

@@ -42,6 +42,7 @@ members = [
  "vm/loader/igvmfilegen",
  "vm/vmgs/vmgs_lib",
  "vm/vmgs/vmgstool",
+  "opentmk"


We can probably start a new 'category' of members here for tmk-related crates. I'm assuming we'll have more in the future.

smalis-msft · 2025-04-22T15:26:58Z

Cargo.toml

@@ -525,6 +526,7 @@ xshell-macros = "0.2"
 # We add the derive feature here since the vast majority of our crates use it.
 #zerocopy = { version = "0.7.32", features = ["derive"]}
 zerocopy = { version = "0.8.14", features = ["derive"]}
+linked_list_allocator = "0.10.5"


smalis-msft · 2025-04-22T15:27:15Z

xtask/src/tasks/guest_test/uefi/gpt_efi_disk.rs

@@ -22,7 +22,7 @@ pub fn create_gpt_efi_disk(out_img: &Path, with_files: &[(&Path, &Path)]) -> Res
        ));
    }

-    let disk_size = 1024 * 1024 * 32; // 32MB disk should be enough for our tests
+    let disk_size = 1024 * 1024 * 512; // 32MB disk should be enough for our tests


Why the increase?

the image size of the compiled application goes beyond 300MB, So we had to increase the disk size.

Should probably make it a parameter then, so that guest-test-uefi doesn't also get bigger.

smalis-msft · 2025-04-22T15:27:28Z

opentmk/README.md

@@ -0,0 +1,3 @@
+# `guest_test_uefi`
+
+See the guide for more info on how to build/run the code in this crate.


I'll make the changes.

smalis-msft · 2025-04-22T15:28:28Z

opentmk/build_deploy.sh

@@ -0,0 +1,3 @@
+RUST_BACKTRACE=1 cargo build -p opentmk --target x86_64-unknown-uefi
+cargo xtask guest-test uefi --bootx64 ./target/x86_64-unknown-uefi/debug/opentmk.efi


This is fine for v0, but eventually we'll probably want to port this and guest-test-uefi construction to an xflowey process.

I'll remove this in this PR and add the xflowey changes in a follow up PR.

smalis-msft · 2025-04-22T15:29:00Z

opentmk/build_deploy.sh

@@ -0,0 +1,3 @@
+RUST_BACKTRACE=1 cargo build -p opentmk --target x86_64-unknown-uefi
+cargo xtask guest-test uefi --bootx64 ./target/x86_64-unknown-uefi/debug/opentmk.efi
+qemu-img convert -f raw -O vhdx ./target/x86_64-unknown-uefi/debug/opentmk.img  ~/projects/opentmk.vhdx


We probably should try to use a fixed vhd1 instead of vhdx, because I think that will be more portable (i.e. will work in hyper-v and openvmm on both windows and linux)

Right. I think thats the plan I will work on it with the xflowey changes.

smalis-msft · 2025-04-22T15:29:41Z

opentmk/Cargo.toml

+edition.workspace = true
+rust-version.workspace = true
+
+[dependencies]


nit: sort and switch to dotted syntax for crates that don't need features. Also move all crates to be workspaced.

@smalis-msft There is one issue where I want to use the serde package with default features off as I want to use a no_std env. Should I disable the feature in workspace?

smalis-msft · 2025-04-22T15:30:10Z

opentmk/src/main.rs

@@ -0,0 +1,27 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+#![allow(warnings)]


This will need to go away eventually

got it. We don't have any warnings, I will remove this.

smalis-msft · 2025-04-22T15:30:37Z

opentmk/src/main.rs

+#![allow(warnings)]
+#![no_std]
+#![allow(unsafe_code)]
+#![feature(naked_functions)]


Is it not possible to do what we're doing wihtout these features? We've been able to stick to stable for everything else so far.

we either need naked functions or the x86 interrupt abi feature.

Looks like naked functions will be stabilized in 1.88 rust-lang/rust#134213, but we'll have to leave that functionality disabled or commented out until then.

smalis-msft · 2025-04-22T15:31:03Z

opentmk/src/main.rs

+#![feature(concat_idents)]
+
+#![doc = include_str!("../README.md")]
+// HACK: workaround for building guest_test_uefi as part of the workspace in CI.


Not guest_test_uefi anymore, applies throughout

smalis-msft · 2025-04-22T15:35:05Z

opentmk/src/slog.rs

@@ -0,0 +1,242 @@
+#![feature(panic_location)]
+#[no_std]


Could all this be done as a tracing or log backend instead? That'd provide us a lot of the definitions here for free without having to define it all ourselves.

I wanted a find control on the structure of the logs as they have to be handled by the test harness, I will see if we can have some way to achieve both of them together.

smalis-msft · 2025-04-22T15:36:40Z

opentmk/src/sync.rs

+/// An unbounded channel implementation with priority send capability.
+/// This implementation works in no_std environments using spin-rs.
+/// It uses a VecDeque as the underlying buffer and ensures non-blocking operations.
+pub struct Channel<T> {


Is there a crate out there that provides this sort of functionality? If not then this should move to its own crate in support/

I would prefer to keep this as an internal module for now and move them to a crate in a follow up once we have more confidence on the APIs stability.

Things in support don't need to be stable, we break them all the time. They're solely for our internal use.

smalis-msft · 2025-04-22T15:38:17Z

opentmk/src/slog.rs

+pub static mut SERIAL: Serial<InstrIoAccess> = Serial::new(InstrIoAccess {});
+
+#[macro_export]
+macro_rules! tmk_assert {


This feels like we're combining two very separate concepts: how to format our output, and how to send that output. They should probably be separated into layers. That could also allow us to someday have multiple different formatters and multiple different output backends and mix n match them if need be.

The slog was supposed to do structuring using the serial device, but I get your point. The APIs can be more cleanly separated.

smalis-msft · 2025-04-22T15:42:31Z

opentmk/src/uefi/init.rs

+
+    let os_loader_indications_result = uefi::runtime::get_variable(
+        os_loader_indications_key,
+        &uefi::runtime::VariableVendor(guid!("610b9e98-c6f6-47f8-8b47-2d2da0d52a91")),


Move the guid to a const to avoid repetition

I'll do that

smalis-msft · 2025-04-22T15:43:00Z

opentmk/src/uefi/init.rs

+    )
+    .expect("Failed to get OsLoaderIndications");
+
+    let _ = unsafe { exit_boot_services(MemoryType::BOOT_SERVICES_DATA) };


Why ignored?

We dont use the memory map returned by the exit_boot_services

I see, ok then in that case we should add a comment and/or a name to this var like _uefi_memory_map

smalis-msft · 2025-04-22T15:45:05Z

opentmk/src/uefi/hypvctx.rs

+}
+
+/// Implementation of the `TestCtxTrait` for the `HvTestCtx` structure, providing
+/// various methods to manage and interact with virtual processors (VPs) and


All of this documentation is great, but it should move to the trait definition and individual methods.

Makes sense. I'll do that

smalis-msft · 2025-04-22T15:48:38Z

opentmk/src/uefi/alloc.rs

+    pub unsafe fn init(&self, size: usize) -> bool {
+        let pages = ((SIZE_1MB * size) / 4096) + 1;
+        let size = pages * 4096;
+        let mem: Result<core::ptr::NonNull<u8>, uefi::Error> = boot::allocate_pages(AllocateType::AnyPages, MemoryType::BOOT_SERVICES_DATA, pages);


Why the two different allocators?

this is there to handler cases where we need to do the allocation before we exit boot services. After exit boot services we cant depend on the uefi allocator and we need to switch.

I see. Is there some way we can make this cleaner? Like funadamentally splitting up these two allocators and switching between them at a higher level?

smalis-msft · 2025-04-22T15:50:08Z

opentmk/src/uefi/tests/hv_processor.rs

+    criticallog, infolog, sync::{self, Mutex}, tmk_assert, uefi::context::{TestCtxTrait, VpExecutor}
+};
+
+pub fn exec(ctx: &mut dyn TestCtxTrait) {


All this is suuuuuper cool.

smalis-msft · 2025-04-22T15:52:35Z

opentmk/src/uefi/tests/mod.rs

+pub mod hv_processor;
+pub mod hv_misc;
+
+pub fn run_test() {


I think what we'll want here is some sort of type definition for a Test that has a string name and other niceties. Then we can have an array of Tests and decide what tests to run and how at runtime.

Right the v1 plan is to have a config file which we can read using uefi APIs. The config can define the test to run.

smalis-msft · 2025-04-22T15:53:58Z

opentmk/src/uefi/tests/hv_processor.rs

+
+                    let vtl = ctx.get_current_vtl();
+                    infolog!("vtl: {:?}", vtl);
+                    tmk_assert!(vtl == Vtl::Vtl1, format!("vtl should be Vtl0 for VP {}", i));


vtl 1, not 0?

i'll fix this.

smalis-msft · 2025-04-22T15:54:23Z

opentmk/src/uefi/tests/hv_processor.rs

+        // Testing VTL0
+        {
+            let (mut tx, mut rx) = crate::sync::Channel::new().split();
+            ctx.start_on_vp(VpExecutor::new(i, Vtl::Vtl0).command(


What would happen if these blocks were switched? i.e. if we tried to run on vtl0 on these other vps before vtl1?

The current underlying implementation lights up VTL1 either way. To simplify the execution.

smalis-msft · 2025-04-22T15:56:55Z

opentmk/src/uefi/hypvctx.rs

+///
+/// - `start_on_vp(&mut self, cmd: VpExecutor)`:
+///   Starts a virtual processor (VP) on a specified VTL. Handles enabling VTLs,
+///   switching between high and low VTLs, and managing VP execution contexts.


Add docs for what happens to a vp after the cmd finishes?

right, so every VP is like an executor engine which receives the commands to run if the command is for a different vp it switches to the other VTL and reruns the command in a new context. there is a busy loop waiting for commands on every VP.

smalis-msft · 2025-04-22T15:57:48Z

opentmk/src/uefi/hypvctx.rs

+///   switching between high and low VTLs, and managing VP execution contexts.
+///
+/// - `queue_command_vp(&mut self, cmd: VpExecutor)`:
+///   Queues a command for a specific VP and VTL.


Add docs for how the queue is drained, and what happens if someone calls start_on_vp for the same vp/vtl combo

Sure. I'll add that.

smalis-msft · 2025-04-22T15:58:12Z

opentmk/src/uefi/hypvctx.rs

+/// - `queue_command_vp(&mut self, cmd: VpExecutor)`:
+///   Queues a command for a specific VP and VTL.
+///
+/// - `switch_to_high_vtl(&mut self)`:


Should this and switch_to_low_vtl just combine into a single switch_to_vtl?

smalis-msft · 2025-04-22T15:58:30Z

opentmk/src/uefi/hypvctx.rs

+///   Switches the current execution context to a low VTL.
+///
+/// - `setup_partition_vtl(&mut self, vtl: Vtl)`:
+///   Configures the partition to enable a specified VTL.


Add docs for what will happen if this is not called

smalis-msft · 2025-04-22T15:59:39Z

opentmk/src/uefi/hypvctx.rs

+/// - `start_running_vp_with_default_context(&mut self, cmd: VpExecutor)`:
+///   Starts a VP with the default execution context.
+///
+/// - `set_default_ctx_to_vp(&mut self, vp_index: u32, vtl: Vtl)`:
+///   Sets the default execution context for a specified VP and VTL.
+///
+/// - `enable_vp_vtl_with_default_context(&mut self, vp_index: u32, vtl: Vtl)`:
+///   Enables a VTL for a specified VP using the default execution context.


What is a default context, and how are these different from the above start_on_vp and the like?

the default context is basically setting register context for a VP (basically setting the rip to a default value, where the code receives commands to run on the vp/vtl), while start on VP sets the context and schedules a command to start running on a vp.

Sounds good, document it all

smalis-msft · 2025-04-22T15:59:59Z

opentmk/src/uefi/hypvctx.rs

+/// - `enable_vp_vtl_with_default_context(&mut self, vp_index: u32, vtl: Vtl)`:
+///   Enables a VTL for a specified VP using the default execution context.
+///
+/// - `set_interupt_idx(&mut self, interrupt_idx: u8, handler: fn())`:


What happens if this is called before setup_interrupt_handler?

If setup is not called this is like a no op, the default handlers will be called as defined in the IDT by the UEFI.

Fair enough, that should be documented.

smalis-msft · 2025-04-22T16:00:45Z

opentmk/src/uefi/hypvctx.rs

+///   Enables a VTL for a specified VP using the default execution context.
+///
+/// - `set_interupt_idx(&mut self, interrupt_idx: u8, handler: fn())`:
+///   Sets an interrupt handler for a specified interrupt index. (x86_64 only)


If a function is only implemented for a certain architecture, you can add a #[cfg] to it in the trait definition so it only exists on that architecture.

I'll work on making this change.

smalis-msft · 2025-04-22T16:01:05Z

opentmk/src/uefi/hypvctx.rs

+/// - `get_vp_count(&self) -> u32`:
+///   Retrieves the number of virtual processors available on the system.
+///
+/// - `get_register(&mut self, reg: u32) -> u128`:


This needs a better type than u32 for the register id.

You are right, we probably need our own enum to create an abstraction to be platform independent. Maybe we can make it happen in V1?

Check out x86defs and see if anything in there would work.

sluck-msft · 2025-04-25T22:04:14Z

opentmk/src/arch/aarch64/hypercall.rs

+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+/// Writes a synthehtic register to tell the hypervisor the OS ID for the boot shim.


Suggested change

/// Writes a synthehtic register to tell the hypervisor the OS ID for the boot shim.

/// Writes a synthetic register to tell the hypervisor the OS ID for the boot shim.

sluck-msft · 2025-04-25T22:07:06Z

opentmk/src/slog.rs

@@ -0,0 +1,242 @@
+#![feature(panic_location)]


Does "slog" refer to serial log? It's sort of an unfortunate file name.

Yeah. I am working on it. Steven suggested to abstract the module into two layers on how we log and the structure of the log.

sluck-msft · 2025-04-25T22:31:37Z

opentmk/src/uefi/hypvctx.rs

+
+const ALIGNMENT: usize = 4096;
+
+type ComandTable =


Suggested change

type ComandTable =

type CommandTable =

sluck-msft · 2025-04-25T23:14:01Z

opentmk/src/uefi/context.rs

+pub trait TestCtxTrait {
+    fn get_vp_count(&self) -> u32;
+    fn get_current_vp(&self) -> u32;
+    fn get_current_vtl(&self) -> Vtl;


Is this (and similar methods, like switching VTLs) for the current VP?

Similar questions for get_register, etc

yes. these APIs are for current VP. (I'll add more documentation to explicitly call that out)

sluck-msft · 2025-04-25T23:14:27Z

opentmk/src/uefi/context.rs

+
+    fn setup_partition_vtl(&mut self, vtl: Vtl);
+    fn setup_interrupt_handler(&mut self);
+    fn set_interupt_idx(&mut self, interrupt_idx: u8, handler: fn());


Suggested change

fn set_interupt_idx(&mut self, interrupt_idx: u8, handler: fn());

fn set_interrupt_idx(&mut self, interrupt_idx: u8, handler: fn());

sluck-msft · 2025-04-26T00:27:09Z

opentmk/src/uefi/hypvctx.rs

+            .get_mut(&vp_index)
+            .unwrap()
+            .push_back((cmd, vtl));
+        if vp_index == self.my_vp_idx && self.hvcall.vtl != vtl {


Does vtl refer to the VTL you want the current VP to be on, or the VTL you want the target vp to run the command on?

This VTL switch is for a case where we schedule a cmd on the current VP. The current context will switch to the target VTL on current VP and execute the command. The command is expected to return to initial VTL by calling HvCall/HvReturn.

sluck-msft · 2025-04-26T00:31:05Z

opentmk/src/uefi/hypercall.rs

+
+        let header = hvdef::hypercall::ModifyVtlProtectionMask {
+            partition_id: hvdef::HV_PARTITION_ID_SELF,
+            map_flags: hvdef::HV_MAP_GPA_PERMISSIONS_NONE,


You could make this an input parameter? And then other functions like apply_vtl2_protections could build on top of it.

Makes sense, I'll work o abstracting this more.

sluck-msft · 2025-04-26T00:37:50Z

opentmk/src/arch/x86_64/interrupt_handler_register.rs

+extern "x86-interrupt" fn no_op(stack_frame: InterruptStackFrame) {}
+
+pub fn register_interrupt_handler(idt: &mut InterruptDescriptorTable) {
+    register_interrupt_handler!(idt, 0, handler_0);


Some of these are architectural (and might also be in x86 defs), it might be nice to include the actual name for them.

sluck-msft · 2025-04-26T00:41:08Z

opentmk/src/uefi/tests/hv_processor.rs

+                    let vtl = ctx.get_current_vtl();
+                    infolog!("vtl: {:?}", vtl);
+                    tmk_assert!(vtl == Vtl::Vtl1, format!("vtl should be Vtl0 for VP {}", i));
+                    tx.send(());


Eventually would be nice to check that the register state is what we expect, but that might be difficult if execution keeps going.

Which registers would we want to assert? We can probably think of how we can do it.

A lot of them, eventually. We need to make sure that we correctly get this shared/private split correct, to the extent that we can:

https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/vsm#virtual-processor-state-isolation

sluck-msft · 2025-04-26T00:47:13Z

opentmk/src/uefi/tests/hv_misc.rs

+            val != 0xAA,
+            "heap memory should not be accessible from vtl0"


Seems like we could eventually use the interrupt handler that you set on 0x30 to check that we did get a memory intercept?

(Pretty cool to all this in action though :))

mayank-microsoft added 4 commits April 21, 2025 18:46

feat: opentmk init

6d32033

feat: opentmk init feat: opentmk init feat: opentmk init feat: opentmk init feat: opentmk init feat: opentmk init feat: init 1 feat: init 2 feat: init 1 feat: opentmk feat: opentmk init 3 feat: opentmk init 4 feat: opentmk init 4

feat: added docs

351199b

refactor(opentmk): interrupt handler doesn't need to save state for a…

3e1673e

…ll the registers

reafctor(opentmk): remove dead code

056bf7d

mayank-microsoft force-pushed the target-opentmk branch from c3f74c0 to 056bf7d Compare April 21, 2025 19:14

mayank-microsoft requested review from smalis-msft and sluck-msft April 21, 2025 19:30

smalis-msft reviewed Apr 22, 2025

View reviewed changes

sluck-msft reviewed Apr 26, 2025

View reviewed changes

mayank-microsoft added 2 commits May 11, 2025 19:51

chore: resolve PR feedback

6d37a84

chore: resolve PR feedback

53e0da2

		@@ -0,0 +1,3 @@
		# `guest_test_uefi`

		See the guide for more info on how to build/run the code in this crate.

		@@ -0,0 +1,3 @@
		RUST_BACKTRACE=1 cargo build -p opentmk --target x86_64-unknown-uefi
		cargo xtask guest-test uefi --bootx64 ./target/x86_64-unknown-uefi/debug/opentmk.efi

	/// Writes a synthehtic register to tell the hypervisor the OS ID for the boot shim.
	/// Writes a synthetic register to tell the hypervisor the OS ID for the boot shim.

	fn set_interupt_idx(&mut self, interrupt_idx: u8, handler: fn());
	fn set_interrupt_idx(&mut self, interrupt_idx: u8, handler: fn());

		val != 0xAA,
		"heap memory should not be accessible from vtl0"

		@@ -0,0 +1,242 @@
		#![feature(panic_location)]
		#[no_std]

feat(opentmk): opentmk framework with first testcase #1210

Are you sure you want to change the base?

feat(opentmk): opentmk framework with first testcase #1210

Conversation

mayank-microsoft commented Apr 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayank-microsoft Apr 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayank-microsoft Apr 28, 2025 •

edited

Loading