Skip to content

Delete files which were removed in source dir#41

Closed
eepifanova wants to merge 28 commits intomainfrom
delete-docs-remaining
Closed

Delete files which were removed in source dir#41
eepifanova wants to merge 28 commits intomainfrom
delete-docs-remaining

Conversation

@eepifanova
Copy link
Copy Markdown
Contributor

Using 'az storage blob sync' to sync the content of the dir

This command is based on azcopy:
https://learn.microsoft.com/en-us/cli/azure/storage/blob?view=azure-cli-latest#az-storage-blob-sync
https://github.com/Azure/azure-storage-azcopy/wiki/azcopy_copy

P.S. It's not recommended to pass more than 10 mil files in one job.

Using 'az storage blob sync' to sync the content of the dir
@eepifanova eepifanova requested a review from a team as a code owner February 26, 2025 16:50
jackbhickey
jackbhickey previously approved these changes Feb 27, 2025
Copy link
Copy Markdown
Contributor

@jackbhickey jackbhickey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - I have a test repo to try out the action as well, will do that now 👍

@eepifanova
Copy link
Copy Markdown
Contributor Author

Sorry for long delay. It was really challenging.

TLDR: the manual remove is the only option for now.

I've inspected various options including:
az storage blob sync - in a preview mode, doesn't support 'auth login' properly
azcopy sync - doesn't work with 'auth login' properly
az storage fs directory - ADLS Gen2 file system options aren't available in the storage

The current workflow seems to be working in PREVIEW (there are lots of debug for now).
I think it can be fixed deeply for previews before updating prod process.

What do you think?

@eepifanova eepifanova closed this Jan 26, 2026
@eepifanova eepifanova deleted the delete-docs-remaining branch January 26, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants