Skip to content

[Feature Request] Decouple read cache size and write buffer size in AbstractBufferedFile #761

Description

@suni72

The Problem
Currently, AbstractBufferedFile uses a single argument, block_size, to control both:

  1. The size of the read cache (passed to cache initialization).
  2. The write buffer size (triggering a flush/upload when the buffer exceeds this size).

This coupling creates a limitation for downstream implementations (like gcsfs) where optimal performance might require different values for reading and writing.

Code References

Motivation / Use Case
In gcsfs.ExtendedGcsFilesystem, we want to set a larger buffer for writes (e.g., 16MB default used by python SDK) to optimize upload throughput. However, for reads, we want to keep the cache size smaller (e.g., the default 5MB) to limit memory usage during random access or with many open files.

Because block_size controls both, we cannot increase the write buffer default without unintentionally increasing the read cache, nor can users configure them independently.

Proposed Solution
We propose adding a new argument to AbstractBufferedFile and GCSFileSystem, such as write_block_size (or write_buffer_size).

  • If the new argument is provided, it controls the write buffer threshold.
  • If it is None, it falls back to block_size to maintain backward compatibility.

Contribution
We are happy to submit a PR to implement this change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions