[Future Work] Promote DirectMemmoveBuffer into Standard Bucket Path

[googlyrahman]

ZonalFile manages concurrency through an in-house buffer that preallocates space, allowing workers to write directly to memory. It uses the ctypes.pythonAPI library to execute GIL-free writes, which avoids unnecessary data copies and prevents the global interpreter lock from creating a bottleneck. We call it DirectMemmoveBuffer as of today.

The standard path, by comparison, relies on b"".join. In this process, each worker generates its own byte object, and a new, combined copy is created when the join operation executes. This overhead limits throughput for standard buckets.

As for why we are addressing this now: we initially held off on integrating this with regional buckets due to concerns from the fsspec maintainers regarding the use of the ctypes API. If we are confident in the stability of our buffer, it is time to promote it to the standard bucket

[shcheklein]

uses the ctypes.pythonAPI

QQ - are we concerned / was it discussed anywhere that ctypes.pythonAPI might not be available on pypy for example? It seems this only change might break its support (and it seems it was working so far).

[googlyrahman]

Thank you for bringing this into our attention, I certainly was not aware of these other python implementations as CPython is kind of industry standard.

Interesting observations:

1. I couldn't find any usage statistics of these implementations on internet, but based on my reading, it seems more than 95% of the users relies on standard CPython implementation.
2. It's also good to note, there are good numbers of libraries which doesn't support these alternative implementation, Examples includes - Pytorch, Tensorflow, Jax, HuggingFace. I think the underlying reason is same that these libraries also depend upon CPython implementation.

Regardless, Promoting this buffer to standard path would make a breaking change for our PyPy-based gcsfs customers, Regarding zonal buckets where we already have ctypes.pythonapi dependency, I think it's fine because we never had a PyPy support since beginning of the launch, so there is no breaking change, it never worked on that platform.

I think we can support these other implementations by adding fallbacks, and eventually promote this buffer to standard buckets. The code should be adjusted to add if-else statements before using ctypes. The performance would of course differ in these other implementations, but that's fine as the usage of these implementations is low at this time.

Here's how fallback needs to be added:

import platform

if platform.python_implementation() == 'CPython':
      use ctypes.PythonAPI for GIL free work
else:
     use standard bytes operation which may be GIL bound in another python implementation

@zhixiangli, @ankitaluthra1 - Do ensure we add these fallbacks before we promote DirectMemmoveBuffer to the standard path, or more specifically the use of ctypes API on standard path.