Feature request: client-side write coalescing / configurable finalize-debounce for hot (repeatedly-rewritten) objects

[chaitanyapantheor]

Note: This is a feature request / enhancement, not a bug. The repository only provides a bug-report issue template, so this follows that template's structure where it applies (System & Version below) and adds the standard feature-request sections.

Is your feature request related to a problem? Please describe.

Applications that repeatedly rewrite a single object — progress/status files, lock files, counters, config-state files — hit the GCS per-object mutation limit (~1 mutation/sec to a single object name) and get:

rpc error: code = ResourceExhausted desc = The object <bucket>/<path> exceeded the rate limit for object mutation operations (create, update, and delete). Please reduce your request rate. See https://cloud.google.com/storage/docs/gcs429.

This is the same error reported in #2783 (labeled known-issues / grpc-known-issues, closed as "retries were added"). Retry-with-backoff smooths bursts, but under sustained churn the object simply cannot accept more than ~1 mutation/sec, so:

• callers that keep retrying block for the full backoff window (and, with ignore-interrupts on, can wedge the FUSE op in uninterruptible state); and
• callers that bound max-retry-attempts (precisely to avoid that wedge) surface the 429 to the application as an I/O error.

Streaming writes only helps when the file is held open and appended sequentially — one finalize on close(). The real-world offenders don't do that: they open(O_TRUNC) / write / close() per tick, or read-back-then-write, each of which finalizes the object (one mutation per tick). No current flag changes that behavior.

Context: we host a large fleet of WordPress/Drupal sites on GCSFuse. Third-party plugins we don't control do this routinely — for example a theme demo-importer that rewrites uploads/<theme>_log/import_progress.txt after every imported item, or Wordfence rewriting wflogs/config-synced.php. We can advise customers, but we can't patch every plugin.

Describe the solution you'd like

An opt-in, client-side write-coalescing / finalize-debounce for hot objects:

• A configurable minimum interval between finalizations of the same object (e.g. write.min-finalize-interval / write.coalesce-window), default off.
• When set, GCSFuse debounces close()/fsync()-driven finalizations on a rapidly-rewritten object so that N rewrites within the window collapse to ≤1 GCS mutation, with a trailing flush after the window drains.
• A clear, documented durability trade-off: within the window the latest bytes are not yet on GCS and not yet visible to other readers — the same trade-off streaming writes already documents for fsync, made explicit and bounded by the window.
• Optionally scoped by a path glob (e.g. only coalesce **/*.tmp, **/wflogs/**, **/*_progress.txt) so it never touches normal writes.

Describe alternatives you've considered

• App-side changes (write transient files to local disk, append-and-hold-open, throttle): correct where we control the code; impossible for third-party plugins.
• Widening retries: re-introduces the wedged/uninterruptible-op behavior; trades an error for a hang.
• write.finalize-file-for-rapid: Rapid-storage-class only; not applicable to standard buckets.

System & Version

• OS: All
• Platform: GKE / GCE VM (applies to any platform)
• Version: confirmed on gcsfuse v3.9.2 (affects v2.0.1 onwards), --client-protocol=grpc, streaming writes default-on

Steps to reproduce

1. Mount any standard (non-Rapid) bucket with --client-protocol=grpc (streaming writes default-on).
2. Repeatedly rewrite a single file faster than once per second, e.g. an open(O_TRUNC) / write / close() loop on one path, or fio re-writing over the same object (as in gRPC: Input/Output error while writing repeatedly over the existing file #2783).
3. Observe ResourceExhausted … exceeded the rate limit for object mutation operations in the logs; with bounded max-retry-attempts this surfaces to the application as an I/O error.

Additional context

• Supersedes the underlying need behind gRPC: Input/Output error while writing repeatedly over the existing file #2783, which reported this error but was closed on retry alone. There is currently no open issue tracking a reduction of the mutation rate itself.