Feature Suggestion: Optimize handling of "Stored" (Uncompressed) entries to avoid unnecessary memory/disk caching

[crashGG]

Hi Dr. Peterson,
First of all, thank you for developing SimpleZipDrive! The current Hybrid Caching Engine is a very smart way to balance RAM usage and performance based on file size.I would like to suggest a minor optimization regarding how the engine handles "Stored" (uncompressed, 0% compression level) files within archives.The IssueCurrently, the engine strictly differentiates files by size. However, users often create large ZIP archives using the "Stored" method solely for convenient file bundling and archiving. Or some encoded files are difficult to compress and using store. For example, mkv, ps3 iso. Since these entries are completely uncompressed, they inherently support byte-level seeking and can be read directly from the source archive. Treating them like compressed files causes redundant overhead:Small Stored Files: Cached in-memory needlessly, consuming RAM.Large Stored Files (≥512 MB): Decompressed/copied to a temporary disk cache, causing severe and unnecessary disk I/O bottlenecks.The SuggestionInstead of relying solely on file size, could the Hybrid Caching Engine first check the compression method of the entry?If an entry is "Stored" (Compression Method == 0), SimpleZipDrive could implement a direct-read/streaming approach via standard file seeking, completely bypassing both the in-memory and temporary disk caches.This would dramatically improve performance and reduce resource consumption when browsing or extracting large archived bundles.Thank you for considering this optimization!
Best regards,crashGG.

[drpetersonfernandes]

OK.
I start working on that.

[drpetersonfernandes]

it is working for zipped file!! Not working for rar files. Bug for 7z files. I will try a litle more.

I will release it today

[crashGG]

Hi again!
First of all, huge thanks for implementing the fast path for stored entries so quickly. The way you leveraged OpenEntryStream() to pin the dataStart offset and wrap it in a custom Stream is exceptionally elegant.That said, while the current implementation guarantees correctness via the global mutex (lock (_sourceLock)), it introduces a significant performance bottleneck due to lock contention and stream pointer thrashing under high-concurrency workloads.Here are two realistic scenarios where the current design introduces severe I/O degradation:Scenario 1: Multi-File Concurrent Reads (Heavy I/O Contention)If a user mounts an archive containing multiple large, uncompressed media files (e.g., two .mkv videos) and attempts to stream or copy them simultaneously, the global lock serializes all read operations. Instead of leveraging the OS file system's native multi-channel capabilities or high-speed SSD parallel throughput, the background threads are forced to run synchronously, destroying concurrent read performance.Scenario 2: Single-File Nested Multi-Threaded Seeking (e.g., Game Resource Archives)Consider a single massive, uncompressed game resource bundle (like an OpenBOR .pak, a 3D engine asset bundle, or a PS3 .iso) mounted via the virtual drive. Game engines frequently fire up multiple threads to concurrently query different byte offsets within that single file (e.g., Thread A reads textures, Thread B pre-loads audio).Because they all hit the single StoredEntryStream instance, the global lock turns this parallel asset-loading pipeline into a single-file bottleneck.More critically, because standard streams are inherently stateful (relying on a mutable shared _position), every thread switch forces the underlying _sourceStream.Position to aggressively re-seek across drastically different offsets. This continuous pointer thrashing bypasses the OS read-ahead cache and cripples performance.The Suggestion: Paradigm Shift to a "Stateless" or "Per-Thread" ArchitectureTo support modern workloads like gaming or multi-stream 4K playback natively from the virtual drive, we should decouple the data-fetching mechanism from the stream's internal state.Instead of relying on a shared stateful stream pointer wrapped in a critical section, we can transition to a Stateless / Concurrent Architecture:Leverage Offset-Based Atomic Reads:Modern .NET runtimes offer native capabilities to perform atomic, thread-safe file reads by passing an absolute offset directly down to the OS kernel. By transitioning from Stream.Read() (which relies on modifying the stream's internal pointer state) to stateless, offset-driven APIs, we can bypass the file pointer mechanics entirely. This allows the OS to handle concurrent disk I/O requests asynchronously at the hardware level.Eliminate the Global Lock:By switching to stateless reads where each request explicitly specifies absoluteOffset = _dataOffset + threadLocalOffset, we no longer risk corrupted read cursors across concurrent threads. The global lock (_sourceLock) can be completely eliminated, enabling true zero-overhead parallel reads.Stateless Virtualization Layer Compatibility:Since the underlying VFS driving engine (like Dokan) operates on a stateless model—where the OS kernel explicitly requests specific byte counts at precise offsets (ReadFile(Offset, Length)) for every incoming filesystem request—making our underlying stream engine stateless natively bridges this architectural gap.By shifting away from stateful synchronization toward an immutable, offset-based read path, SimpleZipDrive would achieve near-native SSD read speeds even under heavy multi-threaded game or file-copying workloads.I sincerely hope these architectural observations are helpful to your project.

[drpetersonfernandes]

https://github.com/drpetersonfernandes/SimpleZipDrive/releases/tag/release_2.1.0

[drpetersonfernandes]

Thank you for all the hints and suggestions.

[crashGG]

Hello, I tested the v2.1.0, and everything worked as expected in stored zip files, but I encountered a performance issue.
I packaged a 15GB file as a stored zip archive and verified its hash value using xxhsum -H3 (this method is the fastest on my Zen4 CPU, eliminating CPU bottlenecks). With original unpacked file, the verification time is 40.2 seconds. After mounting with simplezipdrive, the verification time is 1 minute and 15.3 seconds, which is 87.5% slower than the original file. Theoretically, the speed reduction should be within 10%.

[drpetersonfernandes]

Hi crashGG,

Thank you for testing v2.1.0 and for the detailed performance benchmark.

I did try to improve performance for the stored entry fast path. I experimented with implementing a read-ahead logic to pre-fetch data, and also tried using a significantly bigger read cache. Unfortunately, both attempts failed - the results were either the same or actually slower than the current implementation.

I'll continue investigating, but I wanted to be transparent that these were the approaches I tried and neither yielded improvements. If you have any specific ideas or code samples that might help, I'd be happy to try them.

Do you have a script to perform your test?

Best regards,
Peterson Fernandes

[crashGG]

here is my test cmd script:
@echo off
chcp 65001 >nul 2>&1

:: Capture Start Time (Total Milliseconds)
for /f %%a in ('powershell -Command "[DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds()"') do set "start_ms=%%a"

echo Calculation started, please wait...
echo ------------------------------------------

:: ============================================
:: xxhsum -H3 the fastest hash check on modern CPUs, with vector acceleration.
:: testbig1 a big file stored in zip,mount to R: with SimpleZipDrive
xxhsum -H3 r:\testbig1
:: ============================================

echo ------------------------------------------
:: Capture End Time (Total Milliseconds)
for /f %%a in ('powershell -Command "[DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds()"') do set "end_ms=%%a"

:: Calculate Elapsed Time via PowerShell to prevent CMD 32-bit integer overflow
for /f %%a in ('powershell -Command "%end_ms% - %start_ms%"') do set "total_ms=%%a"

:: Break down total milliseconds into Hours, Minutes, Seconds, and Milliseconds
set /a ms=%total_ms% %% 1000
set /a total_s=%total_ms% / 1000
set /a s=%total_s% %% 60
set /a total_m=%total_s% / 60
set /a m=%total_m% %% 60
set /a h=%total_m% / 60

:: Output the formatted execution time
echo.
echo Elapsed Time: %h%h %m%m %s%s %ms%ms
pause

xxHash:
https://github.com/Cyan4973/xxHash

[crashGG]

Regarding the performance dip on sequential reads (like xxhsum),I have some preliminary thoughts: the root cause is likely that OS-level automated Read-Ahead (Page Cache) gets disabled when strictly use offset-driven RandomAccess.Read. The OS treats it as random I/O, completely missing out on hardware-level streaming speeds.Instead of adding an unstable user-space read cache, could implement an adaptive hybrid approach inside ReadAt. could try a non-blocking Monitor.TryEnter(_sourceLock) to check if the current request is perfectly sequential with _sourceStream.Position. If it is, leverage the native stream to let the OS preload data; if there's lock contention or a seek jump, could fall back to the stateless RandomAccess.Read. This way, could get faster speeds for file copying.
This snippet can be dropped right into the core of ReadAt. It leverages a non-blocking Monitor.TryEnter to achieve an opportunistic fast-path for streaming, while seamlessly falling back to our stateless path without thread blockages.
// Fast-path for single-threaded sequential reads: leverage the stateful stream to trigger native OS read-ahead if there's no contention.
if (Monitor.TryEnter(_sourceLock)) {
try {
if (_sourceStream.Position == _dataOffset + fileOffset)
return _sourceStream.Read(buffer, bufferOffset, maxBytes);
} finally { Monitor.Exit(_sourceLock); }
}
// Fall back to stateless, lock-free RandomAccess under lock contention (concurrency) or seek jumps (random I/O).
return RandomAccess.Read(_fileHandle, buffer.AsSpan(bufferOffset, maxBytes), _dataOffset + fileOffset);
One minor detail:
Please ensure that the underlying zip archive handle/stream isn't opened with FileOptions.RandomAccess. It should be left as default or explicitly set to FileOptions.SequentialScan. If RandomAccess is passed during setup, it forces a persistent hint onto the NT kernel, which would neuter the OS read-ahead optimization even when route sequential requests through the stateful stream via fast-path.

Hope this helps.
Best regards,crashGG.

[drpetersonfernandes]

I just implemented your suggestion.
Look at the results

Direct Access

---

Timestamp: 2026-05-22 16:31:12
File: D:\Samples\Xbox360\ABC Wipeout 2 (USA).iso
Size: 13917.88 MB (13.59 GB)
Bytes read: 14593952402
Read time: 28.762 seconds
Read speed: 483.9 MB/s

Algorithm: XXH3 (xxhsum sequential)
XXH3 Hash: \XXH3_64fa89b1d4c760a2
XXH3 time: 30.21 seconds
XXH3 speed: 460.7 MB/s
Algorithm: XXH3 Random Access
Random XXH3 Hash: XXH3_45ca793196b9e627
Random reads: 1024 blocks of 4096 bytes (4096 KB total)
Random access time: 0.309 seconds
Random access speed: 12.94 MB/s

release_2.1.0

---

Timestamp: 2026-05-22 17:06:18
File: M:\ABC Wipeout 2 (USA).iso
Size: 13917.88 MB (13.59 GB)
Bytes read: 14593952402
Read time: 31.623 seconds
Read speed: 440.12 MB/s

Algorithm: XXH3 (xxhsum sequential)
XXH3 Hash: \XXH3_64fa89b1d4c760a2
XXH3 time: 32.338 seconds
XXH3 speed: 430.39 MB/s
Algorithm: XXH3 Random Access
Random XXH3 Hash: XXH3_78eb53e45a571666
Random reads: 1024 blocks of 4096 bytes (4096 KB total)
Random access time: 0.109 seconds
Random access speed: 36.7 MB/s

release_2.2.0

---

Timestamp: 2026-05-22 16:57:17
File: M:\ABC Wipeout 2 (USA).iso
Size: 13917.88 MB (13.59 GB)
Bytes read: 14593952402
Read time: 31.809 seconds
Read speed: 437.55 MB/s

Algorithm: XXH3 (xxhsum sequential)
XXH3 Hash: \XXH3_64fa89b1d4c760a2
XXH3 time: 32.289 seconds
XXH3 speed: 431.04 MB/s
Algorithm: XXH3 Random Access
Random XXH3 Hash: XXH3_76e3e1d78d795434
Random reads: 1024 blocks of 4096 bytes (4096 KB total)
Random access time: 0.103 seconds
Random access speed: 38.83 MB/s

[drpetersonfernandes]

Performance Comparison

Test file: ABC Wipeout 2 (USA).iso — 13.59 GB (14,593,952,402 bytes), stored (uncompressed) in a ZIP archive.
Mounted via: SimpleZipDrive, drive M:.
Hardware notes: Zen4 CPU, XXH3-128 vector-accelerated hash (CPU not a bottleneck).

---

Raw Sequential Read

	Time	Speed	Overhead vs Direct
Direct (D:)	28.76 s	483.90 MB/s	—
v2.1.0 (M:)	31.62 s	440.12 MB/s	+9.95%
v2.2.0 (M:)	31.81 s	437.55 MB/s	+10.59%

XXH3 Sequential Hash

	Time	Speed	Overhead vs Direct
Direct (D:)	30.21 s	460.70 MB/s	—
v2.1.0 (M:)	32.34 s	430.39 MB/s	+7.04%
v2.2.0 (M:)	32.29 s	431.04 MB/s	+6.88%

XXH3 Random Access (1024 × 4KB blocks)

	Time	Speed
Direct (D:)	0.309 s	12.94 MB/s
v2.1.0 (M:)	0.109 s	36.70 MB/s
v2.2.0 (M:)	0.103 s	38.83 MB/s

---

Random access: counterintuitively faster through the mount

The mounted drive outperforms direct access on random reads by ~3× (38 MB/s vs 13 MB/s). This is likely Dokan's internal buffering absorbing the tiny 4 KB seeks more efficiently than the raw NTFS path handles scattered 4096-byte requests. The hash values differ because random offsets are generated independently each run — only sequential hashes are deterministic.

[drpetersonfernandes]

https://github.com/drpetersonfernandes/SimpleZipDrive/releases/tag/release_2.2.0

[drpetersonfernandes]

I wrote a small console application to perform the tests.
It is inside the repository.

FileBenchmark.csproj

[crashGG]

While these benchmark numbers provide great insight, there is a strong probability that the consecutive test runs are measuring OS-level memory caching rather than actual disk I/O.In Windows, subsequent reads of the same file are heavily accelerated by the Windows Page Cache (Standby List). When a 13.59 GB file is accessed sequentially, the OS aggressively caches the data into unused RAM.This caching behavior explains the anomalies observed in the benchmark:The 3× Random Access Anomaly: The counterintuitive 3× performance boost during random access (38 MB/s vs 13 MB/s) is highly symptomatic of memory-resident data. Instead of hitting the physical disk and incurring hardware seek/controller latency, the 4 KB requests are likely being served directly from the Windows Page Cache. This measures RAM latency rather than Dokan or NTFS efficiency.Determinism vs. Non-determinism: Even with randomized offsets, if the entire 13.59 GB file (or large contiguous chunks of it) already resides in the Standby List from the previous sequential runs, any random 4 KB seek will result in a cache hit, bypassing physical disk I/O entirely.How to Isolate Pure Disk I/O (Cold Testing)To eliminate the masking effect of the Windows Page Cache and obtain true hardware-bound I/O and driver overhead metrics, the system cache must be purged between every single test run.You can achieve a strict "cold test" environment using Microsoft's official Sysinternals utility, RAMMap https://learn.microsoft.com/en-us/sysinternals/downloads/rammap

Recommended Benchmarking Workflow

To ensure a scientifically rigorous and apples-to-apples comparison between Direct (D:), v2.1.0, and v2.2.0, the following pipeline should be enforced:
Drop Caches:
Run RAMMap64.exe -Et
Execute Test Run 1 (e.g., Direct Raw Sequential Read)
Drop Caches:
Run RAMMap64.exe -Et
Execute Test Run 2 (e.g., v2.1.0 Raw Sequential Read)
Drop Caches: Run RAMMap64.exe -Et
Execute Test Run 3 (e.g., v2.2.0 Raw Sequential Read)
By enforcing a cache purge prior to each iteration, you will bypass the Windows file system pre-fetching mechanics and expose the true, un-cached file system overhead of SimpleZipDrive and the underlying Dokan layer.

[drpetersonfernandes]

OK.
I ran the tests again clearing the windows cache before each test.

FileBenchmark Results

Test file: Assassin's Creed III (USA) (En,Fr,Es,Pt).wux (WiiU, 16.37 GB) stored as uncompressed ZIP.

All tests run with Windows Standby List purged before each benchmark (cold cache).

Summary

Metric	Direct (F:)	v2.1.0 (M:)	v2.2.0 (M:)
Seq. Read	90.66s @ 184.95 MB/s	176.83s @ 94.82 MB/s	176.76s @ 94.86 MB/s
XXH3 Seq.	91.03s @ 184.20 MB/s	176.83s @ 94.82 MB/s	176.85s @ 94.81 MB/s
XXH3 Rand.	9.72s @ 0.41 MB/s	9.78s @ 0.41 MB/s	9.88s @ 0.40 MB/s
XXH3 Hash	24f1e7eb6ef6d55e	24f1e7eb6ef6d55e	24f1e7eb6ef6d55e

Key Findings

1. Mounted throughput is 49% of direct — 95 MB/s via SimpleZipDrive vs 185 MB/s direct. Overhead is ~95%, significantly higher than the ~10% seen in the earlier 13.59 GB ISO test.

2. v2.1.0 and v2.2.0 are functionally identical — the stored-entry fast path in v2.2.0 provides no measurable improvement for this file (sub-0.1s difference, within noise).

3. Random access is HDD-bound — all three configurations converge at ~0.4 MB/s for random 4K reads (mechanical seek latency dominates).

Direct Access

---

Timestamp: 2026-05-23 17:25:03
File: F:\Nintendo WiiU\Assassin's Creed III (USA) (En,Fr,Es,Pt).wux
Size: 16767.31 MB (16.37 GB)
Cache: cleared before each test (Windows Standby List purged)
Bytes read: 17581801472
Read time: 90.657 seconds
Read speed: 184.95 MB/s
Algorithm: XXH3 (xxhsum sequential)
XXH3 Hash: \XXH3_24f1e7eb6ef6d55e
XXH3 time: 91.027 seconds
XXH3 speed: 184.2 MB/s
Algorithm: XXH3 Random Access
Random XXH3 Hash: XXH3_1879137a828f93d5
Random reads: 1024 blocks of 4096 bytes (4096 KB total)
Random access time: 9.72 seconds
Random access speed: 0.41 MB/s

Version 2.1

---

Timestamp: 2026-05-23 17:44:04
File: M:\Assassin's Creed III (USA) (En,Fr,Es,Pt).wux
Size: 16767.31 MB (16.37 GB)
Cache: cleared before each test (Windows Standby List purged)
Bytes read: 17581801472
Read time: 176.834 seconds
Read speed: 94.82 MB/s
Algorithm: XXH3 (xxhsum sequential)
XXH3 Hash: \XXH3_24f1e7eb6ef6d55e
XXH3 time: 176.834 seconds
XXH3 speed: 94.82 MB/s
Algorithm: XXH3 Random Access
Random XXH3 Hash: XXH3_bbb4c77933c564b2
Random reads: 1024 blocks of 4096 bytes (4096 KB total)
Random access time: 9.78 seconds
Random access speed: 0.41 MB/s

Version 2.2

---

Timestamp: 2026-05-23 17:52:48
File: M:\Assassin's Creed III (USA) (En,Fr,Es,Pt).wux
Size: 16767.31 MB (16.37 GB)
Cache: cleared before each test (Windows Standby List purged)
Bytes read: 17581801472
Read time: 176.76 seconds
Read speed: 94.86 MB/s
Algorithm: XXH3 (xxhsum sequential)
XXH3 Hash: \XXH3_24f1e7eb6ef6d55e
XXH3 time: 176.853 seconds
XXH3 speed: 94.81 MB/s
Algorithm: XXH3 Random Access
Random XXH3 Hash: XXH3_2862aa883e474adc
Random reads: 1024 blocks of 4096 bytes (4096 KB total)
Random access time: 9.881 seconds
Random access speed: 0.4 MB/s