refactor(blk): offload virtqueue drain to worker thread - #95
Open
agicy wants to merge 8 commits into
Open
Conversation
agicy
force-pushed
the
refactor-blk-worker
branch
2 times, most recently
from
June 9, 2026 12:54
10eebf0 to
f90ae43
Compare
agicy
marked this pull request as ready for review
June 9, 2026 12:56
Add NULL checks for vdev and dev to prevent crashes on misconfigured or partially-initialized devices. Guard close(img_fd) so we only close valid file descriptors.
When open(2) fails, img_fd is -1. Calling close(-1) is harmless (returns EBADF) but semantically wrong. Just remove it.
Extract the queue-empty check into a proper inline function in virtio.h. Uses __atomic_load_n with __ATOMIC_ACQUIRE on avail_ring->idx so that a consumer seeing a new idx also observes all descriptor writes the guest made before writing idx. virtqueue_is_empty() in virtio.c now delegates to vq_is_empty().
Advertise VIRTIO_BLK_F_FLUSH and handle VIRTIO_BLK_T_FLUSH requests via fdatasync(2) on the backing image. This allows the guest to persist previously completed writes to stable storage.
Add pre-allocated iovec arrays to BlkDev so the hot path can avoid malloc/free for descriptor parsing. Add thread_started flag for safe worker lifecycle management. Change close from int to bool. Add start_blk_worker() declaration.
Add blk_do_read, blk_do_write, blk_do_get_id, and blk_complete as static functions encapsulating the per-operation I/O logic and used-ring update. Not wired in yet — the existing blkproc and complete_block_operation are unchanged. These will replace them in the next commit.
Replace the two-phase architecture (parse in main thread, enqueue to procq, dequeue in worker, dispatch) with a single-phase design where the worker thread owns the virtqueue exclusively: - Rewrite virtq_blk_handle_one_request() to parse, validate, dispatch, and complete in one pass using process_descriptor_chain_buf with pre-allocated buffers — zero heap allocations per request. - Rewrite blkproc_thread() to drain the avail_ring directly via the standard virtio disable-notify / process / enable-notify loop. - Simplify notify_handler() to just signal the worker via condvar. - Advance last_avail_idx on descriptor chain parse failure to avoid spinning forever on corrupt descriptor chains. - Validate header and status byte presence before dereferencing. Remove the intermediate TAILQ/procq, struct blkp_req, and the old parsing/dispatch functions no longer needed.
…worker Use calloc with NULL check in init_blk_dev. Check pthread_mutex_init and pthread_cond_init return values with self-cleanup on failure. Split start_blk_worker out of init_blk_dev so the worker thread is only started after the backing image is successfully opened by virtio_blk_init. Add idempotency guard via thread_started flag. Add BLKGETSIZE64 ioctl fallback in virtio_blk_init for real block devices where st_size is zero. Use SECTOR_BSIZE instead of magic 512 constant. Update virtio_blk_close with thread_started guard so we only join a thread that was actually created. Update create_virtio_device in virtio.c to check init_blk_dev return value and call start_blk_worker after virtio_blk_init.
agicy
force-pushed
the
refactor-blk-worker
branch
from
July 31, 2026 09:44
f90ae43 to
9836392
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR refactors the virtio-blk worker threading model to eliminate the intermediate producer-consumer queue between the main thread and the I/O worker thread.
Before
blkp_reqstructs onto a cross-thread queue (procq).procq's mutex for every single I/O request.After
pthread_cond_signal— it never touches the virtqueue.Performance
Test setup: ramdisk backend, fio + io_uring, 120s runtime, RK3588.
Bug Fixes
While refactoring, several pre-existing issues were fixed:
pthread_joinsovirtio_blk_closedoesn't join an uninitialized thread handle if the worker was never started.written_lenon read error: Whenpreadvfails,written_lenno longer carries-1into the used-ring update (was reporting 0 bytes despite the status byte being written).written_lenforGET_ID: Now reports correct data length via the used ring so the guest sees the full device ID string.close(-1)on open failure: Removed spuriousclose(-1)in the error path whenopen()fails.fstatreturnsst_size = 0(real block devices), fall back toioctl(BLKGETSIZE64).