Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tools to better manage git-annex remotes and DDR inventory #243

Open
3 of 5 tasks
gjost opened this issue Sep 19, 2024 · 2 comments
Open
3 of 5 tasks

Add tools to better manage git-annex remotes and DDR inventory #243

gjost opened this issue Sep 19, 2024 · 2 comments
Assignees
Labels
fixed Issue addressed and awaiting closure

Comments

@gjost
Copy link
Member

gjost commented Sep 19, 2024

We discovered that git annex special remotes for backing up our binary files were not as up-to-date as we thought. This was due to a lack of data about the status of DR collection repositories and their remotes, which led to a breakdown of manual backup processes.

  • Add CLI command ddrremote check
  • ddrremote check write script info to log e.g. YYYY-MM-DDTHH:MM:SS ddrremote check ERROR missing hq-backup-gold files/ddr-pc-32-1/files/ddr-pc-32-1-administrative-bd7710419c-a.jpg
  • mod script to skip repos checked recently(?) checked since DATE(?)
  • --mod-script-to-sleep-1-2min-betw-collections-- (NOPE this happens in calling script)
  • ddrremote recap DAYS command to report what repos were modified within DAYS, focusing on repo creation and addition of annex files.
@gjost gjost added the WORKING label Sep 19, 2024
@gjost gjost self-assigned this Sep 19, 2024
@gjost gjost changed the title Add ddrremote command for working with git-annex remotes. Start with checker Add tools to better manage git-annex remotes Oct 22, 2024
@gjost gjost changed the title Add tools to better manage git-annex remotes Add tools to better manage git-annex remotes and the DDR inventory Oct 22, 2024
@gjost gjost changed the title Add tools to better manage git-annex remotes and the DDR inventory Add tools to better manage git-annex remotes and DDR inventory Oct 22, 2024
@gjost
Copy link
Member Author

gjost commented Oct 22, 2024

I ended up adding the following:

ddrremote copy

Wrapper around the git annex copy command that works with both rsync and b2 remotes, and which logs metadata about the copy operations.

ddrinventory report

Gathers data from local repositories, Cgit on mits2, and specified git annes special remotes to report the state of various parts of the Repository.

  • Presence or absence of repositories in the local filesystem.
  • Sync status of local collection repositories' status compared to Cgit.
  • For each special remote:
  • Presence or absence of collections' data in remote.
  • Presence of files added since the last git annex copy.
  • Errors or inconsistencies in the data from copy operations.

@gjost gjost added fixed Issue addressed and awaiting closure and removed WORKING labels Oct 22, 2024
@gjost
Copy link
Member Author

gjost commented Oct 22, 2024

We have a good-enough solution for the time being. Revisit later for something better...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fixed Issue addressed and awaiting closure
Projects
None yet
Development

No branches or pull requests

1 participant