Skip to content

Generate per-container SAS tokens for Azure AD/Managed Identity auth#6902

Open
adamrtalbot wants to merge 13 commits intomasterfrom
gelatinous-barnacle
Open

Generate per-container SAS tokens for Azure AD/Managed Identity auth#6902
adamrtalbot wants to merge 13 commits intomasterfrom
gelatinous-barnacle

Conversation

@adamrtalbot
Copy link
Collaborator

@adamrtalbot adamrtalbot commented Mar 9, 2026 •
edited
Loading

Summary

Fixes #5669

When using Azure Entra ID (Active Directory) or Managed Identity authentication with Azure Batch, Nextflow previously only generated a SAS token for the working directory container. Pipelines referencing blobs in other containers (e.g., `az://igenomes/...` while workdir is `az://scidev-useast/...`) failed with authentication errors.

This PR generates per-container SAS tokens for all containers accessed during a pipeline run.

Changes

Core infrastructure (nf-azure plugin)

  • AzFileSystemProvider: Owns the per-container SAS token cache (ConcurrentHashMap<String,String>) as runtime state; getSasToken(containerName) returns a container-specific token or falls back to the global sasToken; setSasToken() and getContainerSasTokens() allow callers to register and enumerate tokens; generateAndRegisterContainerSas() is called during newFileSystem0() for AD/MI auth, lazily generating SAS for each new container accessed
  • AzStorageOpts: Config-only — per-container SAS state removed; sasToken remains as the global/user-supplied token
  • AzHelper: Added generateUserDelegationKey() and a new generateContainerSasWithActiveDirectory(BlobContainerClient, Duration, UserDelegationKey) overload
  • AzBatchExecutor: Eagerly generates SAS for the workDir container at startup; resolves AzFileSystemProvider from the workDir AzPath to register the token
  • AzFileSystem: copy() uses per-container SAS lookup via the provider
  • AzFileCopyStrategy: Holds a reference to AzFileSystemProvider (resolved from bean.workDir in the constructor); constructor pre-generates SAS for all input file containers; getSasForPath() looks up and lazily generates tokens via the provider; getEnvScript() exports per-container AZ_SAS_<CONTAINER> env vars from the provider's token map
  • AzBatchService: getSasForPath() resolves the provider from the AzPath argument for per-container lookup
  • AzFusionEnv: getOrCreateSasToken() and getEnvironment() resolve the provider from Global.session.workDir; exports AZURE_STORAGE_SAS_TOKEN_<CONTAINER> per container
  • AzPathFactory: getUploadCmd() resolves the provider from the target AzPath for per-container SAS lookup

Bash library (AzBashLib)

  • nxf_az_upload and nxf_az_download now embed SAS tokens in URLs at script-generation time and split the URL at ? using bash parameter expansion (${var%%\?*} / ${var#*\?}) to correctly insert path components before the query string
  • Removed nxf_az_sas() function (no longer needed)

Backward Compatibility

  • Global sasToken (account key auth) is unchanged — getSasToken(containerName) falls back to it when no per-container token exists
  • Per-container logic only activates when sasToken is null (i.e., AD/MI auth)

Testing

  • Unit tests updated: AzBashLibTest, AzFileCopyStrategyTest, AzStorageOptsTest
  • End-to-end tested with nf-canary across 3 separate Azure Blob containers:
    • scidev-useast (workdir)
    • igenomes (foreign input via --remoteFile)
    • outputs (publishDir)
    • All 14 tasks succeeded including cross-container staging, publish, folder round-trips, and bin scripts

@adamrtalbot
Generate per-container SAS tokens for Azure AD/Managed Identity auth
9f2d4bc 
When using Azure Entra (Active Directory) or Managed Identity authentication
with Azure Batch, generate a separate SAS token per blob container accessed
by the pipeline instead of a single global SAS for the working directory only.

This allows pipelines to access blobs from multiple Azure containers (e.g.,
az://scidev-useast/... and az://igenomes/...) without requiring any user
configuration changes.

Changes:
- AzStorageOpts: add per-container SAS token map with getSasToken(container)
  fallback to global sasToken for backward compatibility
- AzHelper: add generateContainerSasWithActiveDirectory overloads accepting
  UserDelegationKey to avoid redundant key fetches
- AzFileSystemProvider: generate and register per-container SAS on newFileSystem0()
  for AD/MI auth
- AzBatchExecutor: generate SAS for workDir container eagerly at startup for AD/MI
- AzFileCopyStrategy: getSasForPath() generates SAS lazily at script-generation
  time; getEnvScript() exports AZ_SAS_<CONTAINER> env vars per container
- AzBatchService: use getSasForPath(task.workDir) instead of global sasToken
- AzBashLib: add nxf_az_sas() for runtime per-container SAS lookup from env vars;
  nxf_az_download directory fallback handles URLs with embedded SAS tokens
- AzFusionEnv: export AZURE_STORAGE_SAS_TOKEN_<CONTAINER> per container
- AzFileSystem: use provider.getSasToken() for per-container SAS on copy

Generated by Claude Code

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@adamrtalbot
Fix per-container SAS token generation and URL construction for uploads
c1fc69b 
- AzFileCopyStrategy: pre-generate SAS tokens for all input file containers
  in the constructor so getEnvScript() exports AZ_SAS_<CONTAINER> for every
  container used by the task (not just the workdir)
- AzBashLib: fix nxf_az_upload to split target URL at '?' before appending
  the filename, preventing malformed URLs like 'container?sas/filename'
- AzBashLib: fix nxf_az_download directory fallback to split source URL at
  '?' before appending '/*', preventing malformed URLs like 'path*?sas'

Generated by Claude Code

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@adamrtalbot
Fixup: thread safety, dead code removal, test updates, remove local c…
3154527 
…onfig

- Use ConcurrentHashMap for containerSasTokens (thread safety)
- Remove dead generateContainerSasIfNeeded() from AzBatchExecutor
- Make uploadCmd() an instance method; add static overload for AzPathFactory
- AzPathFactory.getUploadCmd() looks up per-container SAS from AzConfig
- Simplify AzFusionEnv conditional into unconditional loop
- Update AzFileCopyStrategyTest and AzBashLibTest for URL-splitting bash functions
- Remove accidentally committed local nextflow.config test file

Generated by Claude Code

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@netlify
Copy link

netlify bot commented Mar 9, 2026 •
edited
Loading

✅ Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit aaa53cd
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69afcefda0866a0008acd143

@adamrtalbot adamrtalbot mentioned this pull request Mar 9, 2026
Open
adamrtalbot
adamrtalbot commented Mar 9, 2026
View reviewed changes
plugins/nf-azure/src/main/nextflow/cloud/azure/config/AzStorageOpts.groovy Outdated
* Used when accessing multiple blob containers with AD/MI authentication.
* ConcurrentHashMap ensures safe concurrent reads and writes from multiple task threads.
*/
private final Map<String,String> containerSasTokens = new ConcurrentHashMap<>()
Copy link
Collaborator Author

@adamrtalbot adamrtalbot Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be better as some object rather than Map<String, String> but I think this is sufficient to demonstrate the fix.

@adamrtalbot
Remove synchronized from setSasToken now that ConcurrentHashMap is used
0151b3e 
Generated by Claude Code

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@adamrtalbot
Move per-container SAS token cache from AzStorageOpts to AzFileSystem…
e6fb133 
…Provider

AzStorageOpts is user configuration; runtime-generated SAS tokens belong
in AzFileSystemProvider which owns the filesystem lifecycle. All callers
(AzBatchService, AzBatchExecutor, AzFileCopyStrategy, AzFusionEnv,
AzPathFactory) now resolve the provider from the AzPath they already hold.

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@adamrtalbot
Remove AZ_SAS_* env var exports from batch scripts
fed4cc0 
SAS tokens are already embedded in each URL in the generated script.
Exporting them as env vars was redundant since nxf_az_upload/download
never read them.

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@Skitionek
Copy link

Skitionek commented Mar 9, 2026 •
edited
Loading

@adamrtalbot I am a bit lost why managed identity approach uses SAS token.

https://docs.azure.cn/en-us/entra/identity/managed-identities-azure-resources/tutorial-linux-managed-identities-vm-access?pivots=identity-linux-mi-vm-access-storage
Related azcopy docs:
https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-authorize-managed-identity?tabs=linux

Could you help me understand the motivation of not using managed identity credentials directly?

@adamrtalbot
Copy link
Collaborator Author

adamrtalbot commented Mar 9, 2026

@adamrtalbot I am a bit lost why managed identity approach uses SAS token.

https://docs.azure.cn/en-us/entra/identity/managed-identities-azure-resources/tutorial-linux-managed-identities-vm-access?pivots=identity-linux-mi-vm-access-storage Related azcopy docs: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-authorize-managed-identity?tabs=linux

Could you help me understand the motivation of not using managed identity credentials directly?

The Managed Identity allows Nextflow to authenticate to Azure. The tasks are unaware they are in a Nextflow pipeline and can't inherit this authentication. Therefore, we generate short access tokens so Nextflow can pass authentication to each individual task.

To use Managed Identities in tasks you must:

  1. Add a user assigned managed identity to the pool
  2. Use Fusion
  3. Enable the following setting:

azure.batch.poolIdentityClientId

New in version 25.05.0-edge.

The client ID for an Azure managed identity that is available on all Azure Batch node pools. This identity is used by Fusion to authenticate to Azure storage. If set to 'auto', Fusion will use the first available managed identity.

@adamrtalbot
Unify SAS token generation across all Azure auth modes
d4a3fd4 
All auth modes (account-key, AD/MI, user-supplied SAS) now follow the
same per-container SAS flow via AzFileSystemProvider. No global SAS is
exported to batch scripts; tokens are embedded directly in URLs.

Fusion retains a temporary account-wide SAS for account-key auth until
Fusion adds per-container SAS support (AZURE_STORAGE_SAS_TOKEN_*).

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@adamrtalbot
Fix container SAS generation: use CONTAINER_PERMS only and AzPath.con…
81d1110 
…tainerClient()

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@adamrtalbot
Delete nextflow.config
d75e68a 
Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
@Skitionek
Copy link

Skitionek commented Mar 9, 2026 •
edited
Loading

To use Managed Identities in tasks you must:

  1. Add a user assigned managed identity to the pool
  2. Use Fusion
  3. Enable the following setting:

So we agree that it is possible to access storage from tasks under condition that managed identity is added to the pool.

The Managed Identity allows Nextflow to authenticate to Azure. The tasks are unaware they are in a Nextflow pipeline and can't inherit this authentication. Therefore, we generate short access tokens so Nextflow can pass authentication to each individual task.

Based on above it would be sufficient, if tasks could be given indication to use managed identity instead of SAS token (under a.m. condition).

It could be also considered as a fallback method if SAS token is not provided.

Would love to hear your input about it, however we might be under a risk of going off topic. I believe this PR is right on conceptual level and it patches an issue currently in focus. I will try to find some time to test it as well but cannot make promise on time frame right now.

@adamrtalbot
Misc fixes and cleanup
ad4a057 
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@adamrtalbot
Fix BashWrapperBuilderWithAzTest: update expected bash lib to URL-emb…
ad63b92 
…edded SAS format

The AzBashLib was updated to embed SAS tokens directly in URLs using
target_base/target_qs splitting instead of the old $AZ_SAS env var
approach. BashWrapperBuilderWithAzTest still expected the old format,
causing two test failures in CI.

Generated by Claude Code

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@adamrtalbot
Fix BashWrapperBuilderWithAzTest: use SAS-embedded URL in unstage_out…
c4c4d42 
…puts assertions

AzPathFactory.getUploadCmd() and AzFileCopyStrategy.httpUrl() now embed
the SAS token directly in the URL. The unstage_outputs assertions were
still comparing against the bare URL (no SAS), causing SpockComparisonFailure.

Extract the SAS from the provider at test time and pass it to
AzHelper.toHttpUrl() so the expected value matches the actual output.

Generated by Claude Code

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@adamrtalbot
Merge branch 'master' into gelatinous-barnacle
aaa53cd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

bug executor/azure-batch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Not authenticated to use blobs outside of Azure blob container working directory when using Azure Entra

3 participants

@adamrtalbot @Skitionek @bentsherman