Results of filtering out deleted files.

[andrej-gukov-tg]

Hi, i have a few questions.

1. im running invert-paths, supplying list of files that were deleted, due to the amount of files i need to run filter-repo a few (10±) times. While ammount of commits and objects drops with each loop, git repack takes more and more time each loop, why?
   1a) not sure if thats really the case but it seems that during repack repository grows to x4-5 its size, assuming that it unpacks and repacks i would think its max 2-3 original size ?
2. Does path-deleted-sizes.txt (analyze result) provide files that actually are not in the repo anymore, or using path-all-sizes.txt with set deletion date a better approach ?
3. It seems that git-filter-repo doesnt clean all the garbage after you delete files with no history. Lets say repo (folder) with 1 branch was 100mb, after filter it went to 90mb, force pushing and cloning results in 80mb folder. Whats the catch ?

[newren]

1. As an aside, why do you feel you need to run filter-repo so many times? If it's command line length, why not just use --invert-paths together with --paths-from-file? That should allow you to handle arbitrarily many files in a single run. Anyway, running multiple times should be okay, and sometimes it's easier for people, so that's not a problem, but I'm just curious. Anyway, it's not immediately clear to me why the repository would grow significantly or take significantly longer. Do you have a testcase you can share that demonstrates the issue? The only thing I can think of off the top of my head is that if you have a large commit count and are doing that many rewrites and you use some non-default flag for --replace-refs (or do the rewrites in different copies of the repository each time), then I could see the growing number of replace refs as potentially slowing things down.

2. Your question doesn't make much sense. The analyze result files were created by analyzing the repository -- not outside sources -- and the only way it can find a file is if it is in the repository. So every path in path-deleted-sizes.txt is a file that is in the repository. Same is true for path-all-sizes.txt -- path-deleted-sizes.txt is just the subset of path-all-sizes.txt where the file has a deletion date, so it's totally duplicative information, but some found it handy to have another copy of the file that had been filtered to just the deletions. You haven't said what you're using this for or why, but it seems like you may be straight up trusting that whatever you find in there is stuff you want to remove. You should probably read the warning first in .git/filter-repo/analysis/README about how "is this path deleted" not a question which can be answered with a binary answer. (You can also find a copy of the relevant paragraph of that README over at https://github.com/newren/git-filter-repo/blob/main/git-filter-repo#L2495-L2508).

3. If you filter a repository and then force push it, are you pushing all refs or just some? git clone is known to only grab a subset of refs (namely refs/heads/* and refs/tags/*), unless you take very specific steps to specify additional refspecs. If there are other refs involved in your repository rewrite, cloning only a subset of those would understandably shrink the size of the repo. Whether that's what's happening here or there is something else involved can't be determined with the amount of information you've provided, but running e.g. git show-ref after filtering and checking whether you have any refs outside of heads and tags would help answer whether this might be the issue.

[Gregor959]

I have a similar issue , I ran it 10 times or so and the git directory actually barely shrank .. from 500 mgb to 480 mgb...
After running:
git-filter-repo --analyze
all the files and directories I had removed were still present.
..
I also tested it by running a file does not exist e.g. 'gsfldsfds.fddf'
git-filter-repo --invert-paths --path 'gsfldsfds.fddf' --use-base-name --force
and it still kept itself busy...

git-filter-repo --invert-paths --paths-from-file filesIDontWantAnymore.txt

works much better and shrank the repo down to 25 mg
I copied git-filter-repo and git-filter-repo.py into the directory containing the .git files
and then executing the following in command line
python.exe git-filter-repo --invert-paths --paths-from-file filesIDontWantAnymore.txt

What also worked was:
git-filter-repo --analyze
go into .git /filter-repo/analysis and look what a suitable size to cut of would be for incorrectly added files and then e.g. run
git-filter-repo --strip-blobs-bigger-than 2M
this then shrank it considerably to 70 mgb. still big but getting more manageable.