Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kgo sink: do not back off on certain edge case #761

Merged
merged 1 commit into from
Jul 29, 2024
Merged

kgo sink: do not back off on certain edge case #761

merged 1 commit into from
Jul 29, 2024

Conversation

twmb
Copy link
Owner

@twmb twmb commented Jun 10, 2024

  • Produce request created and about to be issued
  • Metadata request resolves and removes the broker that was about to be sent to, updates leadership for the partition
  • recBuf's sink field is updated
  • The old sink then enters handleReqResp, then eventually handleRetryBatches

Previously,

  • Failed partition triggers a metadata refresh and enters a failed state until the metadata refresh clears the failing state. Because a metadata refresh JUST happened, internally this causes a 5s wait by default

Now,

  • Failed partition notices that it is actually NOW on a different broker than the broker that is handling the failure, and does not back off at all, and actually triggers potentially draining on the new sink once decInflight runs

Closes #746.

@twmb
kgo sink: do not back off on certain edge case
e62b402 
* Produce request created and about to be issued
* Metadata request resolves and removes the broker that was about to be sent to, updates leadership for the partition
* recBuf's `sink` field is updated
* The old sink then enters handleReqResp, then eventually handleRetryBatches

Previously,
* Failed partition triggers a metadata refresh and enters a failed state
  until the metadata refresh clears the failing state. Because a
  metadata refresh JUST happened, internally this causes a 5s wait by
  default

Now,
* Failed partition notices that it is actually NOW on a different broker
  than the broker that is handling the failure, and does not back off at
  all, and actually triggers potentially draining on the new sink once
  decInflight runs

Closes #746.
@twmb twmb mentioned this pull request Jun 10, 2024
Closed
@twmb twmb added the patch label Jun 10, 2024
@twmb twmb merged commit b44e16e into master Jul 29, 2024
8 checks passed
@twmb twmb deleted the 746 branch July 29, 2024 04:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aratz-lasa aratz-lasa aratz-lasa approved these changes

Assignees
No one assigned
Labels
patch
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Produce Latency Spikes Due To Race Condition When Brokers Are Scaled Down
2 participants
@twmb @aratz-lasa