Skip to content

Anonymous auth to GCS #416

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
TomAugspurger opened this issue Apr 8, 2025 · 5 comments · Fixed by #404
Closed

Anonymous auth to GCS #416

TomAugspurger opened this issue Apr 8, 2025 · 5 comments · Fixed by #404

Comments

@TomAugspurger
Copy link

I'm trying to make an anonymous request to a Google Cloud Storage bucket. I'm not too familiar with GCS, but I did find google.auth.credentials.AnonymousCredentials: https://googleapis.dev/python/google-auth/1.7.0/reference/google.auth.credentials.html#google.auth.credentials.AnonymousCredentials. However, the following raises an error:

import obstore
import obstore.auth.google
import google.auth.credentials


def main():
    g_credentials = google.auth.credentials.AnonymousCredentials()
    credential_provider = obstore.auth.google.GoogleCredentialProvider(g_credentials)
    store = obstore.store.GCSStore(
        bucket="weatherbench2",
        prefix="datasets/era5/1959-2023_01_10-full_37-1h-0p25deg-chunk-1.zarr",
        credential_provider=credential_provider)

    xs = store.list_with_delimiter()
    print(next(xs))


if __name__ == "__main__":
    main()

with

Traceback (most recent call last):
  File "torch_dataloader.py", line 19, in <module>
    main()
  File "torch_dataloader.py", line 14, in main
    xs = store.list_with_delimiter()
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/obstore/store.py", line 364, in list_with_delimiter
    return obs.list_with_delimiter(
           ^^^^^^^^^^^^^^^^^^^^^^^^
obstore.exceptions.UnauthenticatedError: The operation lacked valid authentication credentials for path External GCP credential provider: InvalidOperation: Anonymous credentials cannot be refreshed.

Debug source:
Unauthenticated {
    path: "External GCP credential provider",
    source: PyErr {
        type: <class 'google.auth.exceptions.InvalidOperation'>,
        value: InvalidOperation('Anonymous credentials cannot be refreshed.'),
        traceback: Some(
            "Traceback (most recent call last):\n  File \".venv/lib/python3.12/site-packages/obstore/auth/google.py\", line 76, in __call__\n    self.credentials.refresh(self.request)\n  File \.venv/lib/python3.12/site-packages/google/auth/credentials.py\", line 322, in refresh\n    raise exceptions.InvalidOperation(\"Anonymous credentials cannot be refreshed.\")\n",
        ),
    },
}
@kylebarron
Copy link
Member

Well one option is to install from this branch: #404. Once the upstream PR apache/arrow-rs-object-store#322 is merged, we can merge it into main here.

To fix this issue at hand, we probably want to add a check

if isinstance(self.credentials, AnonymousCredentials):
    return {"bearer": ""}

But I'm not sure whether that will actually work in the current behavior of object_store, because it will send the bearer token with every request (except for get) even if the bearer token is "". We really need apache/arrow-rs-object-store#322 to be merged here.

@kylebarron
Copy link
Member

You can install from that branch with

pip install git+https://github.com/developmentseed/obstore@5b7f5b9304edcaed31fccbed088fe89835dd0821#subdirectory=obstore

as long as you have Rust installed on your system.

@kylebarron
Copy link
Member

In particular, with #404 , you can construct the store with

GCSStore("bucket", skip_signature=True)

without needing to use a credential provider

@kylebarron
Copy link
Member

Yeeah I played around with this a little more and I don't think this will be fixed until #404 gets merged (I'll merge it as soon as the upstream object_store PR gets merged)

@kylebarron kylebarron mentioned this issue Apr 9, 2025
Merged
@TomAugspurger
Copy link
Author

Thanks for testing that out! I'll keep an eye on those two.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support anonymous GCS connections developmentseed/obstore
2 participants
@TomAugspurger @kylebarron