Delete git history beyond a certain point

[shahzadmajeed]

Hi, I want to delete all git history beyond a certain point such that:

1. I want to keep all commits from Jan 1, 2020 to TODAY
2. I want to keep some release branches (last 20 releases maybe), master, develop and some other currently active ones such that their parent/child relationship (for git merge-base) is preserved so that, on the new cleaned repo, developers can continue feature work smoothly and without confusion.
3. We don't use tags so I'm not worried about them

Is that possible with git-filter-repo? Could you please summarize on what do I have to do in steps? I really appreciate the help on this

-Shahzad

[me-and]

Here's an approach. It'll be horrendously inefficient, but it (or something like it) should work:

git rev-list --since 2020-01-01 master develop <list of other branches you care about> >commits-to-keep
git filter-repo --commit-callback '
  with open('commits-to-keep', 'rb') as keep_file:
    commits_to_keep = set(line.strip() for line in keep_file)
  commit.parents = [p for p in commit.parents if p in commits_to_keep]
'

This should edit the commits you want to keep to detach them from the commits you don't, so the commits you want to remove will be unreachable and get pruned. For less horrible inefficiency, you could use the filter-branch Python library to do something similar, but without needing to re-parse the list of commits to keep for every commit that gets processed.

Alternatively, you could find the oldest commits you want to keep, and follow the steps in the "Parent rewriting" section of the manual to rewrite them to no longer reference the unwanted parent commits. That's more manual work, but might end up being quicker.

In either case, I'm not sure "developers can continue feature work smoothly and without confusion" is likely to be achievable, per the "Discussion" section of the manual. Git deliberately makes it difficult to change or erase history, particularly for codebases that have multiple people working on them.

[shahzadmajeed]

Hi,
Thank you for the reply.

In either case, I'm not sure "developers can continue feature work smoothly and without confusion" is likely to be achievable

I just want them to find their active branches on new repository and be able to co-relate parent/child branches like normal, thats all.

I do have the oldest commit for cutover so I will take a look at "Parent rewriting" section of the manual. Once again, I appreciate these suggestions.

[shahzadmajeed]

Hi, sorry to bother you again.

I have the oldest commit I want to keep but it is unclear from "Parent rewriting" section that how am I going to keep branches changed from "oldest commit" time to the "newest commit" time. That was the main problem I wanted to try "git-filter-repo" for. Any ideas?

Thanks again.

[me-and]

Ah, there are two "Parent rewriting" sections, which is probably not helping! The bit I'm talking about is this one:

git-filter-repo/Documentation/git-filter-repo.txt

Lines 975 to 998 in e7e2452

	Parent rewriting
	~~~~~~~~~~~~~~~~
	To replace $commit_A with $commit_B (e.g. make all commits which had
	$commit_A as a parent instead have $commit_B for that parent), and
	rewrite history to make it permanent:
	--------------------------------------------------
	git replace $commit_A $commit_B
	git filter-repo --force
	--------------------------------------------------
	To create a new commit with the same contents as $commit_A except with
	different parent(s) and then replace $commit_A with the new commit,
	and rewrite history to make it permanent:
	--------------------------------------------------
	git replace --graft $commit_A $new_parent_or_parents
	git filter-repo --force
	--------------------------------------------------
	The reason to specify --force is two-fold: filter-repo will error out
	if no arguments are specified, and the new graft commit would
	otherwise trigger the not-a-fresh-clone check.

Essentially, for each of the oldest commits you want to keep, run git replace --graft <commit-hash>. That will create a replace ref that looks like a root commit; if you run git log at this point, you should see that being the oldest commit in your history. The replace ref only hides the history, though; to completely remove the history you then need to call git filter-repo --force (the documentation explains why --force is needed here), which will rewrite the repository to make the replace refs permanent.

[shahzadmajeed]

So, I'm trying "Parent rewriting" approach now and confused about what you meant by this in your first comment:

Alternatively, you could find the oldest commits you want to keep, and follow the steps in the "Parent rewriting" section of the manual to rewrite them to no longer reference the unwanted parent commits. That's more manual work, but might end up being quicker.

Do, I have to run git replace --graft $oldest_commit $some_temp_root_parent for every commit I want to keep? or just single the oldest commit that was made after the cutover point?

[me-and]

So, I'm trying "Parent rewriting" approach now and confused about what you meant by this in your first comment:

Alternatively, you could find the oldest commits you want to keep, and follow the steps in the "Parent rewriting" section of the manual to rewrite them to no longer reference the unwanted parent commits. That's more manual work, but might end up being quicker.

Do, I have to run git replace --graft $oldest_commit $some_temp_root_parent for every commit I want to keep? or just single the oldest commit that was made after the cutover point?

Say your repository looks like the below, and you want to remove all history older than the marked cut-off:

A     main
|
C D   feature
|/
E
|\
F G
| |   <-- cut-off
H |
|/
J     root commit

That means you want to replace commits F and G with ones that contain the same code but have no parents. So you'd run git replace --graft $hashOfF and git replace --graft $hashOfG. You don't specify $some_temp_root_parent, because you don't need any parent; you're cutting the history off beyond that point.

After running those git replace commands plus git filter-repo, the repository should look like the below:

A     main
|
C D   feature
|/
E
|\
F G   <- new root commits

[shahzadmajeed]

Yes, I tried both replace --graft $hashOfF and git replace --graft $hashOfF $some_temp_root_parent for each commit that was going to loose a parent due to cutoff. And it looks perfect

Thank you for all the help.

[DilLip-Chowdary-Codes]

@shahzadmajeed can you share the final steps/commands for this usecase?

[shahzadmajeed]

@DilLip-Chowdary-Codes it might be different in your situation but is a high level of what I did. Note that I highly recommend trying these on a backup/clone of your repository and don't do it on original one even for final cutoff (you probably want to archive original repository as is):

# 1. Get a fresh shallowed clone since cutover date & checkout branches that fall within cutover date. Those branches will be carried over to new repo
git clone --no-single-branch --shallow-since=$cutover_date $original_repo_clone $fresh_shallowed_commits_clone_dir_name
cd $fresh_shallowed_commits_clone_dir_name
sh ../git-scripts/checkout_branches_since.sh
cd ../
 
# 2. Create another fresh but local clone from above shallowed clone so that 'git filter-repo' can work
git clone --no-single-branch "file:///$(pwd)/$fresh_shallowed_commits_clone_dir_name" $unshallowed_local_clone_dir_name
cd $unshallowed_local_clone_dir_name
 
# 3. Git Grafting - replace deleted parent commits with new parents so that their children commits are not orphans and have a root
settle_grafted_commits_for_new_root_parent
git filter-repo --force --no-ff --debug
 
# Push repository as a mirror to a new remote
git push --mirror --prune --force $new_reduced_size_repo_url

function settle_grafted_commits_for_new_root_parent {
    orphan_commits=($(cat .git/shallow | awk '!u[$0]++'))
    create_cutover_branch_and_new_root_commit
    new_root_commit=$(git rev-list --max-parents=0 "cutover_branch")
    git switch develop
    for orphan_child in "${orphan_commits[@]}"; do
        git replace --graft $orphan_child $new_root_commit
    done
}