Fix prefetch re-reading already-cached chunks off disk - #10
Merged
Merged
Conversation
_process_request checked whether a chunk needed downloading by calling retrieve_chunk(), which reads the chunk's full content back from disk -- for every prefetch request, not just on-demand reads. Since chunk_size is 8 MB and a FUSE read() call is much smaller, the prefetch window (10 chunks ahead) stayed the same across dozens of consecutive read() calls while the cursor crawled through one chunk, and each of those calls re-triggered a full-chunk disk read for content that was already on disk and never even entered the RAM cache (only "high priority" on-demand reads were cached in RAM). Net effect: reading a large file sequentially through the mount could read roughly an order of magnitude more bytes off local disk than the file actually contains, sustained for the life of the read. Prefetch now uses DiskPersistence.chunk_exists(), a plain existence check, to decide whether a download is needed, and still deliberately does not populate the RAM cache -- the download (network I/O) is what prefetch is for; caching a chunk that may never be read is not.
zeehio
added a commit
that referenced
this pull request
Aug 7, 2026
…hxtw Fix prefetch re-reading already-cached chunks off disk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DataManager._process_request()checked whether a prefetched chunk needed downloading by callingpersistence.retrieve_chunk(), which reads the chunk's full content back from disk — for every prefetch request, not just on-demand reads. Sincechunk_sizeis 8 MB and a FUSEread()call is much smaller, the prefetch window (10 chunks ahead) stays the same across dozens of consecutiveread()calls while the cursor crawls through one chunk, so each of those calls re-triggered a full-chunk disk read for content that was already on disk and never even entered the RAM cache.DiskPersistence.chunk_exists(), a plain existence check with no content read, and prefetch now uses it instead ofretrieve_chunk()to decide whether a download is needed. Prefetch still deliberately does not populate the RAM cache — the download (network I/O) is the point of prefetching; caching a chunk that may never be read is not.Test plan
uv run pytest -q— 389 passed, 26 skipped (skips are live-API tests requiring real Databricks credentials, unaffected by this change)uv run ruff check— cleanuv run mypy fuse4dbricks/fs/data_manager.py fuse4dbricks/storage/persistence.py— clean_process_request, plusDiskPersistence.chunk_exists()directly (including a test that asserts it never reads the file's content)🤖 Generated with Claude Code
https://claude.ai/code/session_019abzZUvVSrhjHUFgZj9L4s
Generated by Claude Code