GCSFuse sometimes fails to start on GKE with error "could not find default credentials"

[timeyr]

Describe the issue

When creating a new Pod with GCSFuse as sidecar container, the container sometimes exits during startup because apparently it is unable fetch its credentials:

Error while mounting gcsfuse: failed to create storage handle using createStorageHandle: go storage client creation failed: while creating http endpoint: while fetching tokenSource: DefaultTokenSource: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information

This is most probably due to a race condition regarding the reachability of the GKE metadata server for newly created Pods, as described here: https://cloud.google.com/kubernetes-engine/docs/troubleshooting/authentication#troubleshoot-timeout

When this happens, the sidecar container is not restarted, and it leaves the Pod in a broken state until it is recreated.

System & Version (please complete the following information):

• OS: cos-117-18613-164-109
• Platform: GKE
• Version: GCSfuse: 2.5.2-gke.0, GKE: 1.31.7-gke.1265000

Steps to reproduce the behavior with following information:

Create a Pod with a GCSFuse sidecar (many times).

[koi8]

Even more frequent to happen when ImageStreaming is enabled on the node pool

[Zebradil]

We see this as well.

[jasonious]

We've encountered this as well