Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Set up DVC/LFS for PAF23 #62

Closed
MaxJa4 opened this issue Nov 16, 2023 · 3 comments · Fixed by #89
Closed

[Feature]: Set up DVC/LFS for PAF23 #62

MaxJa4 opened this issue Nov 16, 2023 · 3 comments · Fixed by #89
Assignees
Labels
infrastructure Docker, Project setup, ...

Comments

@MaxJa4
Copy link
Collaborator

MaxJa4 commented Nov 16, 2023

Description

PAF22-Docs for DVC: Link
Official Docs (about the remote storage): Link
GitHub LFS Docs: Link

DVC was used in PAF22 for storing ML datasets and trained models to iterate/switch faster without commits or PRs and is optimized for large file sizes.

It was used in PAF22 together with Google Drive, where they used a dedicated PAF account (@ll7 has the credentials).

LFS doesn't need Google Drive or similar, it can use GitHub itself (< 2GB per file).

Definition of Done

  • Handling ML datasets & trained models is solved
@MaxJa4 MaxJa4 added the question Further information is requested label Nov 16, 2023
@MaxJa4 MaxJa4 moved this to 📋 Backlog in PAF Project Backlog Nov 16, 2023
@ll7
Copy link
Member

ll7 commented Nov 16, 2023

Additional information: The Google Drive account was created as a dedicated PAF-DVC account. I have the credentials, and you can use the account if you like.

@MaxJa4
Copy link
Collaborator Author

MaxJa4 commented Nov 16, 2023

Great, thanks. Lets discuss this in the next review/planning.
Updated issue description accordingly.

@MaxJa4
Copy link
Collaborator Author

MaxJa4 commented Nov 16, 2023

Hint: Installing DVC with pip install dvc dvc-gdrive like suggested in the PAF22 docs, installs a new DVC version which doesn't work for new repositories (so for us).
Use an older version for now: pip install dvc==3.23.0 dvc-gdrive

@MaxJa4 MaxJa4 moved this from 📋 Backlog to 🔖 Ready in PAF Project Backlog Nov 22, 2023
@MaxJa4 MaxJa4 added infrastructure Docker, Project setup, ... and removed question Further information is requested labels Nov 22, 2023
@MaxJa4 MaxJa4 changed the title [Feature]: Set up DVC for PAF23 [Feature]: Set up LFS for PAF23 Nov 22, 2023
@MaxJa4 MaxJa4 moved this from 🔖 Ready to 🏗 In progress in PAF Project Backlog Nov 22, 2023
@MaxJa4 MaxJa4 changed the title [Feature]: Set up LFS for PAF23 [Feature]: Set up DVC/LFS for PAF23 Nov 23, 2023
@github-project-automation github-project-automation bot moved this from 🏗 In progress to ✅ Done in PAF Project Backlog Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Docker, Project setup, ...
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants