Exponential overflow header buffer #83
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kylebarron
commented
Mar 27, 2025
src/reader.rs
Outdated
Comment on lines
250
to
259
| async fn expand_prefetch(&mut self, range: Range<u64>) -> AsyncTiffResult<Bytes> { | ||
| let existing_buffer_length = self.buffer_length() as u64; | ||
| let additional_fetch = | ||
| (existing_buffer_length as f64).powf(self.overflow_fetch_exponent) as u64; | ||
|
|
||
| // Make sure that we fetch at least the entire desired range | ||
| let new_range = | ||
| existing_buffer_length..range.end.max(existing_buffer_length + additional_fetch); | ||
| let buffer = self.reader.get_metadata_bytes(new_range).await?; | ||
| self.buffers.push(buffer); |
There was a problem hiding this comment.
This is where we request the overflow and add it to the existing buffering (self.buffers.push(buffer))
kylebarron
commented
Mar 27, 2025
src/reader.rs
Outdated
| /// ## Panics | ||
| /// | ||
| /// If the range does not fall completely within the pre-cached buffers. | ||
| fn buffer_slice(&self, range: Range<u64>) -> Bytes { |
There was a problem hiding this comment.
This reads from the existing buffers, handling a cross-buffer merge in case the input request overlaps multiple of self.buffers
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
AsyncFileReaderto use&mut selfinstead of&selfon the trait methods. This is necessary so that the buffer can be mutated when fetching the overflow, while also adding it to the cached buffersBoxinstead ofArcin reader code, now that we need mutability of the reader.AsyncFileReaderdirectly on anything that implementsAsyncReadandAsyncSeek. We don't need a wrapper object now that we use&mut selfourselves.This PR is not made on top of #82, but I think some mix of this and #82 makes sense.