Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds predicate when webhook commit changes #2986

Merged
merged 4 commits into from
Oct 24, 2024

Conversation

0xavi0
Copy link
Contributor

@0xavi0 0xavi0 commented Oct 23, 2024

This adds a new predicate when the webhook commit is changed in the GitRepo. It also filters out Job events for creation/deletion, because they add extra reconcile loops that only increase the possibility of race conditions and that are not required.

The reconciler already knows when a Job is created/deleted because it is the owner and does not need to react upon those 2 events from the cluster.

Reconcile calls before filtering the Job create/delete events:

$ kubectl logs -n cattle-fleet-system  gitjob-947bddbbc-gnmfv | grep "Reconciling GitRepo" | wc -l
      17

Reconcile calls after filtering the Job create/delete events: (same gitrepo)

$ kubectl logs -n cattle-fleet-system  gitjob-947bddbbc-7qmdw | grep "Reconciling GitRepo" | wc -l
      13

Refers to: #2969

This adds a new predicate when the webhook commit is changed in the `GitRepo`.
It also filters out `Job` events for creation/deletion, because they add extra reconcile loops
that only increase the possibility of race conditions and that are not required.

The reconciler already knows when a `Job` is created/deleted because it is the owner and does not
need to react upon those 2 events from the cluster.

Refers to: rancher#2969

Signed-off-by: Xavi Garcia <[email protected]>
@0xavi0 0xavi0 self-assigned this Oct 23, 2024
@0xavi0 0xavi0 added this to the v2.10.0 milestone Oct 23, 2024
@0xavi0 0xavi0 marked this pull request as ready for review October 23, 2024 11:14
@0xavi0 0xavi0 requested a review from a team as a code owner October 23, 2024 11:14
@0xavi0 0xavi0 removed this from the v2.10.0 milestone Oct 23, 2024
It("creates a new Job", func() {
Eventually(func() error {
jobName := names.SafeConcatName(gitRepoName, names.Hex(repo+webhookCommit, 5))
return k8sClient.Get(ctx, types.NamespacedName{Name: jobName, Namespace: gitRepoNamespace}, &job)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If more than one job were created, how would we detect it?

Copy link
Contributor Author

@0xavi0 0xavi0 Oct 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The job name depends on the repopath+commit, so unless the commit changes...we'll have the same job all the time.
We can see from time to time Already Exist errors when trying to re-create the job under race condition circumstances, so only 1 job is possible.
Extra note: the Already exists errors don't mean the reconciler is not working.
If, for whatever reason, 2 reconcile calls are called very close from each other and they get to the code where it checks if the job exists and both get that it doesn't.... both will try to create the job and the fastest one will win, the other one will get the error and exit the reconcile call

@0xavi0 0xavi0 requested a review from weyfonk October 24, 2024 10:04
@0xavi0 0xavi0 requested a review from manno October 24, 2024 12:58
@0xavi0 0xavi0 force-pushed the 2969-webhook-commit-predicate branch from 8a551ca to 994cebb Compare October 24, 2024 13:01
@0xavi0 0xavi0 merged commit 21c0a36 into rancher:main Oct 24, 2024
12 checks passed
0xavi0 added a commit to 0xavi0/fleet that referenced this pull request Oct 24, 2024
* Adds predicate when webhook commit changes

This adds a new predicate when the webhook commit is changed in the `GitRepo`.
It also filters out `Job` events for creation/deletion, because they add extra reconcile loops
that only increase the possibility of race conditions and that are not required.

The reconciler already knows when a `Job` is created/deleted because it is the owner and does not
need to react upon those 2 events from the cluster.

Refers to: rancher#2969

---------

Signed-off-by: Xavi Garcia <[email protected]>
Co-authored-by: Corentin Néau <[email protected]>
Co-authored-by: Mario Manno <[email protected]>
0xavi0 added a commit that referenced this pull request Oct 25, 2024
* Adds predicate when webhook commit changes

This adds a new predicate when the webhook commit is changed in the `GitRepo`.
It also filters out `Job` events for creation/deletion, because they add extra reconcile loops
that only increase the possibility of race conditions and that are not required.

The reconciler already knows when a `Job` is created/deleted because it is the owner and does not
need to react upon those 2 events from the cluster.

Refers to: #2969

---------

Signed-off-by: Xavi Garcia <[email protected]>
Co-authored-by: Corentin Néau <[email protected]>
Co-authored-by: Mario Manno <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants