Delete Development Data

[hhirsch]

This clearly belongs into the raw data repository if at all. Blend files of maps don't need to be there at all since blender is perfectly capable of reading our xml files https://github.com/GlPortal/glportal-data/tree/b6f4a7ed81e9833fe8b64eab30af5743d37b1574/dev

So only the files that are relevant have to be moved to the raw data repository and the rest has to be erased from history. As this operation will change history there has to be a backup to another repository before that is done.

[Pr00bit]

HI there, I am new to open contribution, the task seems interesting. Is there any way I can support you with the issue?

[hhirsch]

@Pr00bit Yes. It is basically just:

• Create a backup-repository
• Create an unaltered backup to another repository
• Delete development data from history
• Force push

Creating the backup-repository now. This is it: https://github.com/GlPortal/data-backup

[Pr00bit]

Not sure how to proceed with : Delete development data from history :/
where I can find that history location?

[hhirsch]

@Pr00bit I am glad to explain. You see there are binary files in our data repository that are taking up a lot of space and they make cloning take long. By deleting them we can improve the speed of cloning but since this is git it is not enough to remove the files and commit it. We need to go back in history and remove the files from history so that they are really gone and stop taking up space.

[Pr00bit]

I see, thanks for explanation. So where to start as I never done this :)

[hhirsch]

There is no one true way to do it. There are several tools and commands.
There is though a huge repository of articles about it on the net like for example
https://rtyley.github.io/bfg-repo-cleaner/

[Pr00bit]

thanks for that.

I am not sure but first would be creation of local copy of repo:
git clone --mirror https://github.com/GlPortal/glPortal.git

[Pr00bit]

Ok, I have local copy. Over 45mb downloaded.
So now I have to delete the unnecessary files together with history :)
If you could describe names of files which should be removed ?:)

[hhirsch]

@Pr00bit It is linked in the description. The dev directory. You can pack the whole thing and supply a download so I can put it up to bintray.

Also don't forget to push the unaltered code to the backup repo I've supplied.

[Pr00bit]

Ok so I have downloaded whole directory to local:
https://github.com/GlPortal/glportal-data/tree/b6f4a7ed81e9833fe8b64eab30af5743d37b1574/dev

I am not sure is that what you wanted :/

any suggestion for next step.. :)

[hhirsch]

@Pr00bit Can you give more information on where you are stuck? The checklist has good information but I will add as many additional bullet points as needed and answer any question that you might have.

[Pr00bit]

thanks for your time

if I understand correctly, I should delete all files from DEV directory and then upload it back to Git repo site ? :)

[Pr00bit]

regarding Create an unaltered backup to another repository - I have already copy of repo on my pc, is this enoguh or should I create one more additional copy?

[hhirsch]

No. The history of the files needs to be deleted. That is quite the difference as I've said before.

@Pr00bit This issue requires you to be comfortable with using git. If you don't feel very comfortable with git, force-pushing or rewriting history then you can still easily work on #66

@Pr00bit As I have said the backup repository is https://github.com/GlPortal/data-backup and you need to create a PR with all the current data commits on there. This involves forking the backup, pushing to your fork of the backup and creating a PR to our backup.

[hhirsch]

@Pr00bit Consider joining our chats. This way we can get to know each other and all the team members can help you out more easily.

[Pr00bit]

OMG :) sounds to much for me....... I barely used GIT so far
Thanks for you assistance, I will fokus on #66 then :)

[Pr00bit]

Sure I would like to join chats with you all, but not sure which chats you mean
Is there any separate chatting area, or you mean chats you chat under issues for example ? :)

[hhirsch]

@Pr00bit This is our chat room https://gitter.im/GlPortal/glPortal I'll be offline, soon. But if you join there on a regular basis you can get to know the other team members and you'll get a feel for what you can do.

[Pr00bit]

Thanks for this, I appreciate this. will look in the chat then

[ghost]

@hhirsch If I try to push the unaltered code to the backup repository that you have supplied, it shows me that I don't have the permission to do so, so do I create a repository and push it to that or am I missing something?

[hhirsch]

@Alpha-Orionis Hello. Thank you for commenting. Currently in order to get that done you'd need to create pull requests. I have just created this branch in glportal-data https://github.com/GlPortal/glportal-data/tree/before-dev as yet another backup.

Make sure to add the history in the backup when you create your PR from your fork of https://github.com/GlPortal/data-backup

[ghost]

Make sure to add the history in the backup when you create your PR from your fork of https://github.com/GlPortal/data-backup

But this repository is empty thus forking is not available for https://github.com/GlPortal/data-backup repository.

[hhirsch]

@Alpha-Orionis No problem. Let me push the backup there quickly. Just a second.

[hhirsch]

@Alpha-Orionis Pushed to backup repository.

[b-parikh]

@hhirsch Hello, I'm new to open source development as well, but I'd like to help out with this issue. Hope I'm not too late! I forked https://github.com/GlPortal/data-backup as you recommended, and set about trying to delete the blender files from commit history with filter-branch. I ended up removing two files within dev/models/Map: n3.blend and n3.blend1. After pushing, I don't see them in my remote or in the dev history so I'm assuming filter-branch worked! However, I'm trying to create a pull request now, but GitHub says there's nothing to compare between my fork and data-backup (entirely different commit histories). Here is my fork: https://github.com/ZipZapZop/data-backup. I'd appreciate it if I got some guidance on this, thanks!

[hhirsch]

@zipzapzop You have to use https://github.com/GlPortal/glportal-data as the base repository for your fork.

[AhmedRedaAmin]

@hhirsch , Well hello , sounds like a fun little GIT excursion , also new to open source contribution here , I am just checking whether or not help is still needed , assuming you might have not updated the issue for a while .

Edit : I am getting ZipZapZop problem , I already forked from https://github.com/GlPortal/glportal-data , it seems that my history isn't completely cleaned , there seems to be traces of commits still and I am working on wiping these files completely , but please do tell us how to proceed here , Do we have to git cherry-pick or in other terms reapply all the commits of your original repository ?

[hhirsch]

@AhmedRedaAmin I've made a mistake classifying this as a simple task this is much easier to do with admin permission to the repo.

Thank you all for your help in this issue. I am working on it now.
I hope to talk to you, soon through irc/gitter/slack or reddit.

[AhmedRedaAmin]

No , thank YOU for the opportunity .

[hhirsch]

Fixed with this code

git filter-branch --tree-filter 'rm -rf dev' --prune-empty HEAD
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
git gc
git push origin master --force

And also tested.

[AhmedRedaAmin]

git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
Could you please explain more about this line ? Especially after the pipe .

[hhirsch]

@AhmedRedaAmin If I understand correctly this command removes the reference to the removed files in all the tags and branches. Before I used it the files where gone from master but git clone file transfer was still as big.