HADOOP-19406. ABFS: [FNSOverBlob] Support User Delegation SAS for FNS Blob #7523
Conversation
Test Results============================================================
|
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
| @@ -250,6 +251,7 @@ public void initialize(URI uri, Configuration configuration) | |||
| try { | |||
| if (abfsConfiguration.getAuthType(abfsConfiguration.getAccountName()) == AuthType.SAS && // Auth type is SAS | |||
| !tryGetIsNamespaceEnabled(new TracingContext(initFSTracingContext)) && // Account is FNS | |||
| abfsConfiguration.getFsConfiguredServiceType() == DFS && // Service type is DFS | |||
There was a problem hiding this comment.
Use the constant for DFS
| @@ -75,11 +77,16 @@ public String getDelegationSAS(String accountName, String containerName, String | |||
| case SASTokenProvider.GET_STATUS_OPERATION: | |||
| sp = "e"; | |||
| break; | |||
| case SASTokenProvider.LIST_OPERATION_BLOB: | |||
| sp = "l"; | |||
| sr="c"; | |||
| final URL url = createRequestUrl(destination, | ||
| abfsUriQueryBuilder.toString()); | ||
| final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders(); | ||
| final AbfsRestOperation successOp = getSuccessOp( | ||
| AbfsRestOperationType.RenamePath, HTTP_METHOD_PUT, | ||
| url, requestHeaders); | ||
| successOp.setMask(); |
There was a problem hiding this comment.
Why is this performed only in the rename case ? By default we mask for all cases if auth type is SAS
| @@ -91,14 +91,17 @@ public ITestAzureBlobFileSystemDelegationSAS() throws Exception { | |||
| public void setup() throws Exception { | |||
| isHNSEnabled = this.getConfiguration().getBoolean( | |||
| TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT, false); | |||
| Assume.assumeTrue(isHNSEnabled); | |||
| if(!isHNSEnabled){ | |||
There was a problem hiding this comment.
Add more tests around user delegation SAS specific to FNS Blob and see it works correctly for all cases of implicit, explicit, root path, container path
| @@ -663,13 +663,13 @@ To know more about how SAS Authentication works refer to | |||
| [Grant limited access to Azure Storage resources using shared access signatures (SAS)](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) | |||
|
|
|||
| There are three types of SAS supported by Azure Storage: | |||
| - [User Delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas): Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts. It is Identity based SAS that works at blob/directory level) | |||
| - [User Delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas): Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts or HNS-Disabled Blob Storage accounts. It is Identity based SAS that works at blob/directory level) | |||
There was a problem hiding this comment.
This recommended line should say HNS only
There was a problem hiding this comment.
Yes recommended only for HNS-DFS.
Supported on HNS-DFS and FNS-BLOB
Not supported on FNS-DFS
This should be clearly conveyed
| - [Service SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-service-sas): Global and works at container level. | ||
| - [Account SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-account-sas): Global and works at account level. | ||
|
|
||
| #### Known Issues With SAS | ||
| - SAS Based Authentication works only with HNS Enabled ADLS Gen2 Accounts which | ||
| is a recommended account type to be used with ABFS. | ||
| - SAS Based Authentication works with HNS Enabled ADLS Gen2 Accounts (which |
There was a problem hiding this comment.
Does SAS work correctly with FNS DFS ? We should say FNS here only if we confirm the behaviour for both the endpoints
| @@ -737,7 +737,7 @@ the following configurations apart from above two: | |||
|
|
|||
| - **Security**: More secure than Shared Key and allows granting limited access | |||
| to data without exposing the access key. Recommended to be used only with HNS Enabled, | |||
| ADLS Gen 2 storage accounts. | |||
| ADLS Gen 2 storage accounts or HNS-Disabled Blob Storage accounts. | |||
There was a problem hiding this comment.
again remove non HNS for recommended
| @@ -364,7 +364,7 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur | |||
| abfsUriQueryBuilder.addQuery(QUERY_PARAM_DELIMITER, FORWARD_SLASH); | |||
| } | |||
| abfsUriQueryBuilder.addQuery(QUERY_PARAM_MAX_RESULTS, String.valueOf(listMaxResults)); | |||
| appendSASTokenToQuery(relativePath, SASTokenProvider.FIXED_SAS_STORE_OPERATION, abfsUriQueryBuilder); | |||
| appendSASTokenToQuery(relativePath, SASTokenProvider.LIST_OPERATION_BLOB, abfsUriQueryBuilder); | |||
There was a problem hiding this comment.
Why do we need different SASTokenProvider Operation for listing on Blob? Are permissions different than that of DFS?
| @@ -250,6 +251,7 @@ public void initialize(URI uri, Configuration configuration) | |||
| try { | |||
There was a problem hiding this comment.
Need to update the comment above.
| @@ -62,7 +62,7 @@ public final class AbfsErrors { | |||
| /** | |||
| * Exception message on filesystem init if token-provider-auth-type configs are provided | |||
| */ | |||
| public static final String UNAUTHORIZED_SAS = "Incorrect SAS token provider configured for non-hierarchical namespace account."; | |||
| public static final String UNAUTHORIZED_SAS = "Incorrect SAS token provider configured for non-hierarchical namespace account with DFS service type."; | |||
| @@ -663,13 +663,13 @@ To know more about how SAS Authentication works refer to | |||
| [Grant limited access to Azure Storage resources using shared access signatures (SAS)](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) | |||
|
|
|||
| There are three types of SAS supported by Azure Storage: | |||
| - [User Delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas): Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts. It is Identity based SAS that works at blob/directory level) | |||
| - [User Delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas): Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts or HNS-Disabled Blob Storage accounts. It is Identity based SAS that works at blob/directory level) | |||
There was a problem hiding this comment.
Yes recommended only for HNS-DFS.
Supported on HNS-DFS and FNS-BLOB
Not supported on FNS-DFS
This should be clearly conveyed
| @@ -217,6 +220,7 @@ public void testReadAndWrite() throws Exception { | |||
|
|
|||
| @Test | |||
| public void checkExceptionForRenameOverwrites() throws Exception { | |||
| assumeHnsEnabled(); | |||
Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-19406
In ABFS Driver, user delegation SAS is currently only supported by hierarchical namespace (HNS) enabled accounts (over DFS endpoint). This PR adds support for User Delegation SAS for non-HNS accounts over Blob endpoint as well.
Refer: https://hadoop.apache.org/docs/current/hadoop-azure/abfs.html#Shared_Access_Signature_.28SAS.29_Token_Provider
Tests added in comments below.