Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python] How to use fs.FileSystem.from_uri with Azurite #44682

Open
fabricebaranski opened this issue Nov 8, 2024 · 4 comments
Open

[Python] How to use fs.FileSystem.from_uri with Azurite #44682

fabricebaranski opened this issue Nov 8, 2024 · 4 comments
Labels
Component: Python Type: usage Issue is a user question

Comments

@fabricebaranski
Copy link

Describe the usage question you have. Please include as many useful details as possible.

I want to use fs.FileSystem.from_uri with azurite.
Here a short sample of code:

import pyarrow.fs as fs
local_new, path_new = fs.FileSystem.from_uri("abfs://myaccount.blob.core.windows.net/mycontainer")
local_new.create_dir("test")

The error I received is:
az: error: unrecognized arguments: --scope https://storage.azure.com/.default
How can I set account_key to be able to use FileSystem.from_uri?

Component(s)

Python

@fabricebaranski fabricebaranski added the Type: usage Issue is a user question label Nov 8, 2024
@kou kou changed the title How to use fs.FileSystem.from_uri with Azurite [Python] How to use fs.FileSystem.from_uri with Azurite Nov 8, 2024
@kou
Copy link
Member

kou commented Nov 8, 2024

The feature was removed by #44220.

Could you use an environment variable?
AZURE_PASSWORD may work: https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python

BTW, the error message is strange. Our Azure file system implementation doesn't use the az command.

@fabricebaranski
Copy link
Author

I had to install az because otherwise I received an error because az was not here.
I tried by setting AZURE_PASSWORD, AZURE_STORAGE_ACCOUNT or AZURE_STORAGE_KEY (https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-data-operations-cli#set-environment-variables-for-authorization-parameters) but it didn't work.
I tried also setting query parameter credential_kind to environment but I got

terminate called after throwing an instance of 'Azure::Core::Credentials::AuthenticationException'
  what():  EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Aborted

@fabricebaranski
Copy link
Author

Do you have any example to test FileSystem.from_uri with Azurite?

@fabricebaranski
Copy link
Author

fabricebaranski commented Nov 12, 2024

I tested using fs.AzureFileSystem and it works. But I need to use from_uri so I don't know how to use it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Python Type: usage Issue is a user question
Projects
None yet
Development

No branches or pull requests

2 participants