HADOOP-19736: ABFS. Support for new auth type: User-bound SAS #8051
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
| AUG_03_2023("2023-08-03"), | ||
| NOV_04_2024("2024-11-04"); | ||
| NOV_04_2024("2024-11-04"), | ||
| JULY_05_2025("2025-07-05"); |
Contributor
There was a problem hiding this comment.
We should follow the same format: JUL_05_2025, what do you think?
| abfsClientContext); | ||
| } | ||
|
|
||
| public AbfsClientHandler(final URL baseUrl, |
Contributor
There was a problem hiding this comment.
Java doc missing for the constructor
| this.sasTokenProvider = sasTokenProvider; | ||
| } | ||
|
|
||
| public AbfsClient(final URL baseUrl, |
| encryptionContextProvider, abfsClientContext, AbfsServiceType.DFS); | ||
| } | ||
|
|
||
| public AbfsDfsClient(final URL baseUrl, |
| case UserboundSASWithOAuth: | ||
| httpOperation.setRequestProperty(HttpHeaderConfigurations.AUTHORIZATION, | ||
| client.getAccessToken()); | ||
| httpOperation.setMaskForSAS(); //mask sig/oid from url for logs |
Contributor
Author
There was a problem hiding this comment.
sig stands for signature
| tokenProvider, sasTokenProvider, encryptionContextProvider, | ||
| populateAbfsClientContext()); | ||
| } | ||
| else if (tokenProvider != null) { |
|
|
||
| LOG.trace("Initializing AbfsClient for {}", baseUrl); | ||
| if (tokenProvider != null) { | ||
| if(tokenProvider != null && sasTokenProvider != null){ |
Contributor
There was a problem hiding this comment.
Space between if and (
| 3. Deployed in-Azure with the Azure VMs providing OAuth 2.0 tokens to the application, "Managed Instance". | ||
| 4. Using Shared Access Signature (SAS) tokens provided by a custom implementation of the SASTokenProvider interface. | ||
| 5. By directly configuring a fixed Shared Access Signature (SAS) token in the account configuration settings files. | ||
| 6. Using user-bound SAS auth type, which is requires OAuth 2.0 setup (point 2 above) and SAS setup (point 4 above) |
Contributor
There was a problem hiding this comment.
Grammatical mistake: which requires or which is required?
|
|
||
| public static final String FS_AZURE_TEST_APP_SERVICE_PRINCIPAL_OBJECT_ID = "fs.azure.test.app.service.principal.object.id"; | ||
|
|
||
| public static final String FS_AZURE_END_USER_TENANT_ID = "fs.azure.test.end.user.tenant.id"; |
Contributor
There was a problem hiding this comment.
Rename the variable to FS_AZURE_TEST_END_USER_TENANT_ID
Contributor
There was a problem hiding this comment.
I think TEST should come first. Check for how other test related configs are defined. TestConfigurationKeys.java have them
| public static final String FS_AZURE_TEST_APP_SERVICE_PRINCIPAL_OBJECT_ID = "fs.azure.test.app.service.principal.object.id"; | ||
|
|
||
| public static final String FS_AZURE_END_USER_TENANT_ID = "fs.azure.test.end.user.tenant.id"; | ||
| public static final String FS_AZURE_END_USER_OBJECT_ID = "fs.azure.test.end.user.object.id"; |
| Dec19("2019-12-12"), | ||
| Feb20("2020-02-10"); | ||
| Feb20("2020-02-10"), | ||
| July5("2025-07-05"); |
Contributor
There was a problem hiding this comment.
Same here should be JUL
| private final String sduoid; | ||
|
|
||
| public DelegationSASGenerator(byte[] userDelegationKey, String skoid, String sktid, String skt, String ske, String skv) { | ||
| public DelegationSASGenerator(byte[] userDelegationKey, String skoid, String sktid, String skt, String ske, String skv, String skdutid, String sduoid) { |
Contributor
There was a problem hiding this comment.
add javadoc for what do all these params signify
| qb.addQuery("skv", skv); | ||
|
|
||
| //skdutid and sduoid are required for user bound SAS only | ||
| if(!Objects.equals(skdutid, EMPTY_STRING)){ |
|
|
||
| String stringToSign = sb.toString(); | ||
| LOG.debug("Delegation SAS stringToSign: " + stringToSign.replace("\n", ".")); | ||
| System.out.println("Delegation SAS stringToSign: " + stringToSign.replace("\n", ".")); |
|
|
||
| // Invokes the AAD v2.0 authentication endpoint with a client credentials grant to get an | ||
| // access token. See https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow. | ||
| private String getAuthorizationHeader(String accountName, String appID, String appSecret, String sktid) throws IOException { |
Contributor
There was a problem hiding this comment.
include params in javadoc as well
| // Invokes the AAD v2.0 authentication endpoint with a client credentials grant to get an | ||
| // access token. See https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow. | ||
| private String getAuthorizationHeader(String accountName, String appID, String appSecret, String sktid) throws IOException { | ||
| String authEndPoint = String.format("https://login.microsoftonline.com/%s/oauth2/v2.0/token", sktid); |
Contributor
There was a problem hiding this comment.
Add a constant for the string
| return "Bearer " + provider.getToken().getAccessToken(); | ||
| } | ||
|
|
||
| private byte[] getUserDelegationKey(String accountName, String appID, String appSecret, |
| private byte[] getUserDelegationKey(String accountName, String appID, String appSecret, | ||
| String sktid, String skt, String ske, String skv, String skdutid) throws IOException { | ||
|
|
||
| String method = "POST"; |
Contributor
There was a problem hiding this comment.
we have constants for HTTP methods, can be used here
| final StringBuilder sb = new StringBuilder(128); | ||
| sb.append("https://"); | ||
| sb.append(account); | ||
| sb.append(".blob.core.windows.net/?restype=service&comp=userdelegationkey"); |
Contributor
There was a problem hiding this comment.
Try to use constants as much as possible
| requestBody.append(skdutid); | ||
| requestBody.append("</DelegatedUserTid></KeyInfo>"); | ||
|
|
||
| // requestBody.append("<?xml version=\"1.0\" encoding=\"utf-8\"?><KeyInfo><Start>"); |
anujmodi2021
requested changes
Nov 3, 2025
|
|
||
| public static ApiVersion getCurrentVersion() { | ||
| return NOV_04_2024; | ||
| return JULY_05_2025; |
Contributor
There was a problem hiding this comment.
Are we bumping up version for all the requests?
Wasn't this version bump only needed for Auth related APIs?
Contributor
Author
There was a problem hiding this comment.
Right- we're already overriding the API version header according to skv param for UBS requests. We dont need it here- removed
| this.sasTokenProvider = sasTokenProvider; | ||
| } | ||
|
|
||
| public AbfsClient(final URL baseUrl, |
Contributor
There was a problem hiding this comment.
Should we simply combine the 3 constructors to accept both AccessTokenProvider, SASTokenProvider and the caller can set what it has and null as other?
Contributor
Author
There was a problem hiding this comment.
Makes sense, added
Contributor
There was a problem hiding this comment.
Similar we should do for AbfsBlobClient and AbfsDfsClient
| abfsClientContext); | ||
| } | ||
|
|
||
| public AbfsClientHandler(final URL baseUrl, |
Contributor
There was a problem hiding this comment.
Same as above. We can combine into a single constructor.
But do check if there are any caveats there or a risk of running into NPE
| final AbfsClientContext abfsClientContext) throws IOException { | ||
| URL dfsUrl = changeUrlFromBlobToDfs(baseUrl); | ||
| if (tokenProvider != null) { | ||
| if (tokenProvider != null && sasTokenProvider != null) { |
Contributor
There was a problem hiding this comment.
Apart from UDS, can there be a case where caller can send both as not null? Makes ure no flow leads to this and also add a test around it.
Contributor
Author
There was a problem hiding this comment.
We are initialising both the token providers only for UBS auth type. Added a test for token provider expectations (null or non-null) according to the auth type
| encryptionContextProvider, abfsClientContext, AbfsServiceType.DFS); | ||
| } | ||
|
|
||
| public AbfsDfsClient(final URL baseUrl, |
Contributor
There was a problem hiding this comment.
Is this only for DFS Client?
Contributor
Author
There was a problem hiding this comment.
Validated for blob endpoint sometime back only, missed that. Added now
| * @return sasTokenProvider object based on configurations provided | ||
| * @throws AzureBlobFileSystemException | ||
| */ | ||
| public SASTokenProvider getSASTokenProviderForUserBoundSAS() throws AzureBlobFileSystemException { |
Contributor
There was a problem hiding this comment.
nit: Can be renamed as getUserBoundSASTokenProvider
| getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE, | ||
| null, SASTokenProvider.class); | ||
|
|
||
| if (customSasTokenProviderImplementation == null) { |
Contributor
There was a problem hiding this comment.
Similar check for OAuth also needed right?
For UBS, Cx must configure OAuth as well?
Contributor
Author
There was a problem hiding this comment.
We were getting OAuth token provider separately. Added a single method to get both now
| LOG.trace("Fetching SAS Token Provider"); | ||
| sasTokenProvider = abfsConfiguration.getSASTokenProvider(); | ||
| } else { | ||
| } else if(authType == AuthType.UserboundSASWithOAuth){ |
Contributor
There was a problem hiding this comment.
Here also, it can be combined into a single call with both passed.
Contributor
Author
There was a problem hiding this comment.
Added a method to get both together
|
|
||
| public static final String FS_AZURE_TEST_APP_SERVICE_PRINCIPAL_OBJECT_ID = "fs.azure.test.app.service.principal.object.id"; | ||
|
|
||
| public static final String FS_AZURE_END_USER_TENANT_ID = "fs.azure.test.end.user.tenant.id"; |
Contributor
There was a problem hiding this comment.
I think TEST should come first. Check for how other test related configs are defined. TestConfigurationKeys.java have them
manika137
commented
Nov 7, 2025
| final AbfsClientContext abfsClientContext) throws IOException { | ||
| URL dfsUrl = changeUrlFromBlobToDfs(baseUrl); | ||
| if (tokenProvider != null) { | ||
| if (tokenProvider != null && sasTokenProvider != null) { |
Contributor
Author
There was a problem hiding this comment.
All these conditions can again be removed by creating a single constructor for AbfsDfsClient and AbfsBlobClient that sends both access token provider and sas token provider (like we did for client handler and parent constructors)
But would it be better to have it this way to separate out the logging here?
Contributor
There was a problem hiding this comment.
Logging as well can be combined. in log we will see null value for the one which is not supposed to be used and we will know.
Better to combine everywhere IMO.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
| .split(AbfsHttpConstants.COMMA))); | ||
| } | ||
|
|
||
| public AbfsBlobClient(final URL baseUrl, |
Contributor
There was a problem hiding this comment.
Similarly here also we can do how we did for AbfsClient.
We have 3 constructors here and the only diff they have is the token provider.
Let's combine them and caller can choose what to send
| this.sasTokenProvider = sasTokenProvider; | ||
| } | ||
|
|
||
| public AbfsClient(final URL baseUrl, |
Contributor
There was a problem hiding this comment.
Similar we should do for AbfsBlobClient and AbfsDfsClient
| final AbfsClientContext abfsClientContext) throws IOException { | ||
| URL dfsUrl = changeUrlFromBlobToDfs(baseUrl); | ||
| if (tokenProvider != null) { | ||
| if (tokenProvider != null && sasTokenProvider != null) { |
Contributor
There was a problem hiding this comment.
Logging as well can be combined. in log we will see null value for the one which is not supposed to be used and we will know.
Better to combine everywhere IMO.
manika137
force-pushed
the
HADOOP-19736_UBS
branch
from
85f1a30 to
ec2c76c
Compare
|
💔 -1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-19736
Adding support for new authentication type: user bound SAS
How was this patch tested?
Test suite will be run for the patch