Fix ZooKeeper Topology connection locks not being cleaned up correctly #15757
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
frouioui
requested review from
deepthi,
mattlord,
ajm188,
harshit-gangal,
rohit-nayak-ps and
systay
as code owners
Review ChecklistHello reviewers! 👋 Please follow this checklist when reviewing this Pull Request. General
Tests
Documentation
New flags
If a workflow is added or modified:
Backward compatibility
|
vitess-bot
bot
added
NeedsBackportReason
frouioui
added
Type: Bug
Backport to: release-17.0
and removed
NeedsDescriptionUpdate
frouioui
added
Component: Topology
Backport to: release-18.0
Backport to: release-19.0
|
This PR will unblock the dependencies upgrade Pull Request where this issue has been found (#15743) |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #15757 +/- ##
==========================================
- Coverage 68.40% 68.39% -0.01%
==========================================
Files 1556 1556
Lines 195418 195422 +4
==========================================
- Hits 133666 133663 -3
- Misses 61752 61759 +7 ☔ View full report in Codecov by Sentry. |
mattlord
approved these changes
Apr 19, 2024
go/vt/topo/zk2topo/lock.go
Outdated
| cleanupCtx, cancel := context.WithTimeout(context.Background(), baseTimeout) | ||
| defer cancel() | ||
|
|
||
| if err := zs.conn.Delete(cleanupCtx, nodePath, -1); err != nil { | ||
| log.Warningf("Failed to close connection :%v", err) |
There was a problem hiding this comment.
This error should be changed. It would have been more helpful in understanding the problem if it was:
Suggested change
| log.Warningf("Failed to close connection :%v", err) | |
| log.Warningf("Failed to cleanup unsuccessful lock path %s: %v", nodePath, err) |
There was a problem hiding this comment.
I also think this would be slightly better for line 94:
errToReturn = vterrors.Wrapf(err, "failed to obtain lock: %v", nodePath)
And line 98:
log.Warningf("Failed to obtain lock: %v", err)
There was a problem hiding this comment.
It should be:
log.Warningf("Failed to cleanup unsuccessful lock path %s: %v", nodePath, err)
deepthi
approved these changes
Apr 19, 2024
|
@frouioui please resolve the feedback and then go ahead and merge this. Nice work! |
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
This was referenced Apr 20, 2024
frouioui
added a commit
that referenced
this pull request
Apr 22, 2024
#15757) Signed-off-by: Florent Poinsard <[email protected]>
frouioui
added a commit
that referenced
this pull request
Apr 22, 2024
#15757) Signed-off-by: Florent Poinsard <[email protected]>
frouioui
added a commit
that referenced
this pull request
Apr 22, 2024
#15757) Signed-off-by: Florent Poinsard <[email protected]>
frouioui
added a commit
that referenced
this pull request
Apr 22, 2024
…ned up correctly (#15757) (#15764) Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: Florent Poinsard <[email protected]> Co-authored-by: Florent Poinsard <[email protected]>
frouioui
added a commit
that referenced
this pull request
Apr 22, 2024
…ned up correctly (#15757) (#15762) Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: Florent Poinsard <[email protected]>
frouioui
added a commit
that referenced
this pull request
Apr 22, 2024
…ned up correctly (#15757) (#15763) Signed-off-by: Florent Poinsard <[email protected]> Co-authored-by: Florent Poinsard <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces a new context with timeout to do the clean up phase after we have failed to acquire a new lock in zktopo to prevent the issue described in #15756. Refer to the issue for full context on the issue.
This issue exists on all release branches, but is not visible right now as it is hidden by a bug in the version of
golang.org/x/syncthat we depend on. While it is not strictly necessary to backport to all release branches right now, it is recommended that we do so in order to avoid surprises should we need to upgrade the dependency on a release branch for other reasons.Related Issue(s)