The Problem
Currently, AbstractBufferedFile uses a single argument, block_size, to control both:
- The size of the read cache (passed to cache initialization).
- The write buffer size (triggering a flush/upload when the buffer exceeds this size).
This coupling creates a limitation for downstream implementations (like gcsfs) where optimal performance might require different values for reading and writing.
Code References
Motivation / Use Case
In gcsfs.ExtendedGcsFilesystem, we want to set a larger buffer for writes (e.g., 16MB default used by python SDK) to optimize upload throughput. However, for reads, we want to keep the cache size smaller (e.g., the default 5MB) to limit memory usage during random access or with many open files.
Because block_size controls both, we cannot increase the write buffer default without unintentionally increasing the read cache, nor can users configure them independently.
Proposed Solution
We propose adding a new argument to AbstractBufferedFile and GCSFileSystem, such as write_block_size (or write_buffer_size).
- If the new argument is provided, it controls the write buffer threshold.
- If it is
None, it falls back to block_size to maintain backward compatibility.
Contribution
We are happy to submit a PR to implement this change.
The Problem
Currently,
AbstractBufferedFileuses a single argument,block_size, to control both:This coupling creates a limitation for downstream implementations (like
gcsfs) where optimal performance might require different values for reading and writing.Code References
block_sizeis passed to initialize the read cache here:https://github.com/fsspec/filesystem_spec/blob/e12aa7571244f6695264c92c4867978fed5ad092/fsspec/spec.py#L1919
block_sizeis used to check if the write buffer is full here:https://github.com/fsspec/filesystem_spec/blob/e12aa7571244f6695264c92c4867978fed5ad092/fsspec/spec.py#L2033
Motivation / Use Case
In
gcsfs.ExtendedGcsFilesystem, we want to set a larger buffer for writes (e.g., 16MB default used by python SDK) to optimize upload throughput. However, for reads, we want to keep the cache size smaller (e.g., the default 5MB) to limit memory usage during random access or with many open files.Because
block_sizecontrols both, we cannot increase the write buffer default without unintentionally increasing the read cache, nor can users configure them independently.Proposed Solution
We propose adding a new argument to
AbstractBufferedFileandGCSFileSystem, such aswrite_block_size(orwrite_buffer_size).None, it falls back toblock_sizeto maintain backward compatibility.Contribution
We are happy to submit a PR to implement this change.