Windowed listings can poison the stat cache and cause false `ENOENT` for existing implicit directories

[nk412]

Describe the issue

When gcsfuse mounts a flat-namespace bucket with implicit directories and the negative metadata cache enabled, an empty windowed ListObjects result can overwrite an existing positive stat-cache entry for the directory with a negative entry.

For the duration of --metadata-cache-negative-ttl-secs (5 seconds by default), gcsfuse can then return ENOENT for a directory that still exists, as well as for paths beneath it. The result is served from the local cache without a Cloud Storage request, so there may be no corresponding failed request in Cloud Storage logs or metrics. The directory becomes visible again after the negative entry expires.

The metadata prefetcher provides a concrete path to this state. Once a directory is classified as large, later prefetches use the looked-up child name as ListObjectsRequest.StartOffset. If that name sorts after all existing children, Cloud Storage correctly returns an empty listing for that window. fastStatBucket.insertListing currently treats the empty result as proof that the entire prefix does not exist and calls AddNegativeEntry for the directory.

This appears to affect the negative stat-cache implementation introduced by PR #4729 in commit d830c5c77, released in v3.11.0. I reproduced it on unpatched commit 446c9ea04 and observed the production symptom with gcsfuse 3.11.2.

User-visible impact

• Existing directories transiently return ENOENT.
• All lookups beneath the affected directory can fail during the negative-cache window.
• Multiple processes using the same mount can fail at approximately the same time.
• The condition normally self-heals after the negative TTL, making it appear intermittent.
• Because the rejection can occur entirely in the cache, the failure may have no matching Cloud Storage request or server-side error.

Relevant configuration

The production observation used a mount equivalent to:

gcsfuse --implicit-dirs BUCKET_NAME /path/to/mountpoint

All metadata-cache settings were left at their defaults, which means:

• Negative metadata caching enabled (--metadata-cache-negative-ttl-secs=5)
• Metadata prefetch enabled (--enable-metadata-prefetch=true)
• --metadata-prefetch-entries-limit=5000

The bucket uses a flat namespace, and the affected directory contained approximately 6,400 children — above the default --metadata-prefetch-entries-limit=5000, which is what causes the directory to be classified as large and subsequent prefetches to be windowed.

Setting --metadata-cache-negative-ttl-secs=0 avoided the observed failures under otherwise equivalent load.

System and version

• OS: Linux
• Platform: GCE VM
• Version: gcsfuse 3.11.2
• Bucket type: flat namespace (non-HNS)
• Also reproduced deterministically at the storage/cache layer on commit 446c9ea04

Steps to reproduce

TRACE logs were not captured during the production event; the deterministic reproducer below substitutes for them. It reproduces the cache corruption without requiring a FUSE mount or access to a live bucket, using the real fastStatBucket, the real LRU/stat-cache implementation, and the repository's fake Cloud Storage backend.

Add this test to internal/storage/caching/integration_test.go on an unpatched checkout:

func TestIntegration_WindowedListingDoesNotPoisonExistingDirectory(t *testing.T) {
	deps := setupIntegrationTest(t)
	const dirName = "dir/"

	_, err := storageutil.CreateObject(
		deps.ctx,
		deps.wrapped,
		dirName+"a",
		[]byte("x"),
	)
	require.NoError(t, err)

	// Populate a positive cache entry for the existing implicit directory.
	listing, err := deps.bucket.ListObjects(deps.ctx, &gcs.ListObjectsRequest{
		Prefix:    dirName,
		Delimiter: "/",
	})
	require.NoError(t, err)
	require.NotEmpty(t, listing.MinObjects)

	statReq := &gcs.StatObjectRequest{
		Name:               dirName,
		FetchOnlyFromCache: true,
	}

	dir, _, err := deps.bucket.StatObject(deps.ctx, statReq)
	require.NoError(t, err)
	require.NotNil(t, dir)

	// Model a metadata-prefetch listing whose window begins after the final
	// child. The empty result describes only this window, not the whole prefix.
	listing, err = deps.bucket.ListObjects(deps.ctx, &gcs.ListObjectsRequest{
		Prefix:      dirName,
		Delimiter:   "/",
		StartOffset: dirName + "zzz",
	})
	require.NoError(t, err)
	require.Empty(t, listing.MinObjects)
	require.Empty(t, listing.CollapsedRuns)
	require.Empty(t, listing.ContinuationToken)

	// The implicit directory still exists and should remain positively cached.
	dir, _, err = deps.bucket.StatObject(deps.ctx, statReq)
	assert.NoError(t, err)
	assert.NotNil(t, dir)
}

Run only the reproducer using the repository's Go version in Docker:

docker run --rm \
  --mount type=bind,src="$PWD",dst=/workspace \
  -w /workspace \
  golang:1.26.5 \
  go test ./internal/storage/caching \
    -run TestIntegration_WindowedListingDoesNotPoisonExistingDirectory \
    -count=1 -v

Actual behavior

On the unpatched implementation, the final lookup fails because the empty windowed listing replaced the positive dir/ entry with a negative entry:

gcs.NotFoundError: negative cache entry for dir/

The test therefore fails at the final assert.NoError and assert.NotNil checks even though dir/a still exists in the wrapped bucket.

Expected behavior

An empty windowed or continuation-page listing should not create a negative cache entry for the whole prefix. The existing positive entry for dir/ should remain available, and the test should pass.

Root cause analysis

fastStatBucket.insertListing currently uses the absence of objects and collapsed runs as sufficient evidence that a non-root directory does not exist:

isNegativeCacheEnabled := b.negativeCacheTTL > 0 && !b.enableEmptyManagedFolders
isEmptyNonRootDir := !dirHasContents && dirName != ""
if isNegativeCacheEnabled && isEmptyNonRootDir {
	b.cache.AddNegativeEntry(dirName, b.clock.Now().Add(b.negativeCacheTTL))
}

That inference is sound only when the listing is authoritative for the complete prefix. It is not sound when:

• ListObjectsRequest.StartOffset is set, because the response covers only the namespace at or after that offset;
• the request has a continuation token, because it is not the first page; or
• the response has a continuation token, because the listing is incomplete.

AddNegativeEntry intentionally overwrites an existing cache entry (implementation). Consequently, an empty result for only part of the namespace can replace a valid positive implicit-directory entry.

The metadata-prefetch path makes this reachable in normal operation:

1. A prefetch exceeding metadata-prefetch-entries-limit marks the directory as large.
2. A later uncached child lookup triggers the prefetcher, which builds a windowed listing request using that child name as StartOffset.
3. If the name sorts after every existing child, the windowed listing is empty.
4. insertListing writes a negative entry for the entire directory prefix.
5. If the alternative file-form key is also negative, dirInode.LookUpChild returns not found from cache without contacting Cloud Storage.
6. The false result persists until the negative-cache TTL expires.

The StartOffset behavior is documented by the Cloud Storage Objects list API.

Production observation

This was observed with a directory containing approximately 6,400 children while 13 processes created small files at an aggregate rate of approximately 55 files per second. All 13 processes received ENOENT from open(..., O_CREAT) at approximately the same time. The mount recovered without intervention within several seconds.

Two additional same-day observations on the same prefix included a recursive tree walk failing and a directory listing returning a child whose subsequent open returned ENOENT. Cloud Storage metrics showed no corresponding service-side error responses for the relevant window. Disabling negative metadata caching eliminated the failures under equivalent load.

Suggested resolution

Only derive a negative directory entry from an empty listing when the result is authoritative for the complete prefix. For example, require all of the following before calling AddNegativeEntry:

req.StartOffset == "" &&
req.ContinuationToken == "" &&
listing.ContinuationToken == ""

Note that insertListing currently receives only the listing and directory name, so this guard requires plumbing the *gcs.ListObjectsRequest through from ListObjects into insertListing.

An unwindowed request returning no contents and no continuation token remains valid evidence that the prefix is empty. MaxResults does not require a separate guard in that case: if zero results are returned without a continuation token, the cap did not truncate any results.

I can submit a focused PR with the change and regression coverage if maintainers agree with this direction.

Additional context

A closely matching failure mode was flagged by the automated review (gemini-code-assist) on the original negative stat-cache PR: an empty paginated listing could overwrite a valid positive entry and cause ENOENT.

The concern was acknowledged by the PR author, and the listing-derived negative insertion was removed during that review.

The merged implementation again creates negative entries from empty listings, but does not distinguish complete-prefix listings from windowed or paginated listings.

No duplicate public issue was found when searching for combinations of ENOENT, implicit directories, negative stat cache, metadata prefetch, StartOffset, and empty listings.

[meet2mky]

Thank you for opening the issue. I will update here once we review the PR #5025