Extremely slow cloning when using large SemVer range

[ba-work]

Some info to set the stage:

1. We use a monorepo style structure
2. GitRepositories do not support parsing and extracting tags like ImagePolicies
   • you can parse+extract SemVer in OCI tags like my-app-v1.2.3 => 1.2.3
   • you must use exclusively plain SemVer tags for GitRepositories
3. To get around this deficiency, we've allocated chunks of major SemVer git tags to projects in the repo.
   • project-1 gets 0.0.0 -> 999999.x.x
   • project-2 gets 1000000.0.0 -> 1999999.x.x
   • ...

This means when a team wants to deploy a dev version of kustomizations that control their app, it ends up looking something like this:

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: project-1-dev
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    semver: '>=1000000 <2000000'
  secretRef:
    name: example-ssh-key
  url: ssh://[email protected]/owner/monorepo.git

This has been working very well for a long time, but there is a huge performance problem when the SemVar range is that big. I want to be clear, projects do not actually use that many tags (as writing, the most I see is ~60 majors).

Here is a graph of the enormous performance discrepancy:

In the graph you can see that dev cloning is 50-75x slower and this project in particular only has 49 tags. On top of this it also causes relatively high CPU usage from the source-controller pod.

Potential solutions (in order of preference)

1. Allow SemVer pattern extractions like ImagePolicies. This way we don't have to (ab)use git tags like this.
2. Figure out why it takes so long to compare SemVer tags. I looked around in fluxcd/pkg/git but nothing in the cloning code stood out to me.