Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete files which were removed in source dir #41

Open
wants to merge 28 commits into
base: main
Choose a base branch
from

Conversation

eepifanova
Copy link

Using 'az storage blob sync' to sync the content of the dir

This command is based on azcopy:
https://learn.microsoft.com/en-us/cli/azure/storage/blob?view=azure-cli-latest#az-storage-blob-sync
https://github.com/Azure/azure-storage-azcopy/wiki/azcopy_copy

P.S. It's not recommended to pass more than 10 mil files in one job.

Using 'az storage blob sync' to sync the content of the dir
@eepifanova eepifanova requested a review from a team as a code owner February 26, 2025 16:50
nginx-jack
nginx-jack previously approved these changes Feb 27, 2025
Copy link
Collaborator

@nginx-jack nginx-jack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - I have a test repo to try out the action as well, will do that now 👍

@eepifanova
Copy link
Author

Sorry for long delay. It was really challenging.

TLDR: the manual remove is the only option for now.

I've inspected various options including:
az storage blob sync - in a preview mode, doesn't support 'auth login' properly
azcopy sync - doesn't work with 'auth login' properly
az storage fs directory - ADLS Gen2 file system options aren't available in the storage

The current workflow seems to be working in PREVIEW (there are lots of debug for now).
I think it can be fixed deeply for previews before updating prod process.

What do you think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants