How to combine multiple repositories in a new repository?

[weyert]

I am planning to use git-filter-repo to merge a few multiple repositories into a brand new repository to ease the development. Only I am having some problems with it, looks like I can only merge repository at a time.

For this task I have written a script which is listed below but its not working fully correctly, I am getting the following error when running the following commands:

# Run as:
# git init
# git remote add repo1 https://github.com/weyert/repo1
# git remote add repo2 https://github.com/weyert/repo2
# ./script.sh repo1:projects/repo1:master repo2:projects/repo2:master

This results in the following:

Computing commit graph generation numbers: 100% (721/721), done.
error: pathspec 'master' did not match any file(s) known to git
hint: 'master' matched more than one remote tracking branch.
hint: We found 2 remotes with a reference that matched. So we fell back
hint: on trying to resolve the argument as a path, but failed there too!
hint: 
hint: If you meant to check out a remote tracking branch on, e.g. 'origin',
hint: you can do so by fully qualifying the name with the --track option:
hint: 
hint:     git checkout --track origin/<name>
hint: 
hint: If you'd like to always have checkouts of an ambiguous <name> prefer
hint: one remote, e.g. the 'origin' remote, consider setting
hint: checkout.defaultRemote=origin in your config.
Merging refs: 76cb356e44dc2352bc2b0b9f80e838f98c828fd4 978638a4b619e48f0a3745d53c1bb0a9baf531d2
fatal: Can merge only exactly one commit into empty head

If I try to do the same in sequence by merging on repo at a time instead of trying to do them all in once then the history of the my new target repository is getting staggered while I would like that to be combined together when possible. Would this be possible somehow? I had hope to circumvent this problem by doing the merge of all in one go but no luck.

Help would be appreciated, thank you in advance 🤗 I have to admit this starting to get out of my Git comfort zone 😔

# merge_repos.sh <remote-name>:[:<subdirectory>]:[:<branch>]
for PARAM in $@; do
    # Parse parameters in format <remote-name>[:<subdirectory>]:[:<branch>]
    PARAM_ARR=(${PARAM//:/ })
    REMOTE=${PARAM_ARR[0]}
    SUBDIRECTORY=${PARAM_ARR[1]}
    if [ "$SUBDIRECTORY" == "" ]; then
        SUBDIRECTORY=$REMOTE
    fi
    SOURCE_BRANCH=${PARAM_ARR[2]}
    if [ "$SOURCE_BRANCH" == "" ]; then
        SOURCE_BRANCH="master"
    fi
    REPOSITORY="$(git config --get "remote.$REMOTE.url")"

    # Rewrite all branches from the first remote, only master branches from others
    echo "Building branch '$SOURCE_BRANCH' of the remote '$REMOTE' into directory '$SUBDIRECTORY'"
    REPO_DIR_TMP="$(mktemp -d -t "${REMOTE##*/}.XXXX")"
    echo "Repository url: $REPOSITORY"
    echo "Temporary directory: $REPO_DIR_TMP"

    echo "Checking out branch..."
    git clone --branch $SOURCE_BRANCH "$REPOSITORY" "$REPO_DIR_TMP"

    echo "Filtering branch..."
    pushd $REPO_DIR_TMP
    git filter-repo --to-subdirectory-filter "$SUBDIRECTORY"
    popd

    git remote rm "$REMOTE"
    git remote add "$REMOTE" "$REPO_DIR_TMP"
    git fetch "$REMOTE"

    if [ "$PARAM" == "$1" ]; then
        MERGE_REFS='master'
    else
      MERGE_REFS="$MERGE_REFS $(git rev-parse $REMOTE/$SOURCE_BRANCH)"
    fi
done

# Merge all master branches
COMMIT_MSG="chore: merge repositories"
git checkout master
echo "Merging refs: $MERGE_REFS"
git merge --no-commit -q $MERGE_REFS --allow-unrelated-histories

[weyert]

Anyone any idea, what I could be doing wrong?

[newren]

You might want to use
MERGE_REFS="$REMOTE/master"
so that the branch to merge is unambiguous. Otherwise when you run your checkout command:
git checkout master

Then git complains to you because you have no 'master' branch, but you do have a origin1/master remote tracking branch AND an origin2/master remote tracking branch AND an origin4/master remote tracking branch, etc., and it has no idea which one to use.

However, using your current strategy you'll end up with a history like this:

A---E---F---J---L  feature1
                 \
B---G------------M  master
     ^-feature2  /
C---D---H---I---K feature3

Instead of one like this:

A---B---C---D---E---F---G---H---I---J---K---L master

Which seems to go against what you wanted (" the history of the my new target repository is getting staggered while I would like that to be combined together when possible") If every branch has linear history, and there's an easy way to interleave the commits (e.g. by commit date), then such interleaving could be automated. But if commit dates can't be trusted (e.g. what if one commit has an earlier commit date than one of its parents on one of the lines of history?), or your initial branches contain non-linear history (i.e. you've got merge commits), then how is an interleaver supposed to work? It's a pretty hard problem in general, with no perfect solution. I've not tackled that problem in filter-repo. There are a number of other requests in the issues for something to "combine" histories with some kind of interleaving, but nothing exists currently. Sorry.

[weyert]

Thank you for taking the time to answer my question. So far I haven't had much luck with my little script yet but you are right that I would be prefer the A---B---C---D---E---F---G---H---I---J---K---L master format as you mentioning. Personally, I think it would be by commit date would be great. I will give it a shot again.

[neurocline]

This is something I want as well (I'm turning a bunch of individual repositories into a monorepo). I think what I'm going to do is use git-filter-repo to join everything together in the layout I want, and once that looks good, I'll do a final step of git-fast-export reorder-in-python git-fast-import. I'm pretty familiar with the git-fast-import file format from a project I did about 4 years ago (svnrdump -> git-fast-import -> git, hours to convert large depots instead of weeks/months). I trust the timestamps in the commits of the individual repositories in question. If I do this, I'll post the code somewhere.

[weyert]

Awesome, that would be great, if you could share it! Looking forward seeing it.

[weyert]

@neurocline Did you ever get a chance to have a look at it?

[ngbrown]

https://github.com/mbitsnbites/git-tools/blob/master/join-git-repos.py does this.