dekaf: Upstream kafka fallback and per-connection-type routing #1911
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jan 31, 2025
jshearer
force-pushed
the
jshearer/dekaf_multi_upstream_brokers
branch
from
da51ef0 to
36928fd
Compare
jshearer
force-pushed
the
jshearer/dekaf_materialization_endpoint_support
branch
2 times, most recently
from
af74dfa to
8a6b5b0
Compare
jshearer
force-pushed
the
jshearer/dekaf_multi_upstream_brokers
branch
from
36928fd to
a15d847
Compare
jshearer
force-pushed
the
jshearer/dekaf_materialization_endpoint_support
branch
from
8a6b5b0 to
9a9e9c4
Compare
jshearer
force-pushed
the
jshearer/dekaf_multi_upstream_brokers
branch
from
a15d847 to
440e8a6
Compare
… a Session and write them to the correct ops logs journal Also support filtering logs by the requested shard log level
jshearer
force-pushed
the
jshearer/dekaf_materialization_endpoint_support
branch
from
9a9e9c4 to
3462798
Compare
jshearer
force-pushed
the
jshearer/dekaf_multi_upstream_brokers
branch
from
440e8a6 to
dc394c4
Compare
Then implement some tests to validate field selection logic
jshearer
force-pushed
the
jshearer/dekaf_materialization_endpoint_support
branch
from
3462798 to
d1c933d
Compare
Part of dekaf: Improvements to handle higher scale #1876, we want to implement broker fallback so Dekaf can connect to any of the brokers in the cluster if one doesn't respond. An improvement here would be to periodically fetch the metadata from at least one of the responding brokers and update this list of addresses so that future sessions can know about/use any newly created members of the cluster. I don't anticipate changing the topology of our cluster that frequently, and if we do then updating Dekaf's deployment configs isn't that big of a deal. I may eat my hat on this, we'll see. In addition, we want to move people over to the new MSK cluster, so this implements routing new-style connections to a separate cluster with separate credentials.
jshearer
force-pushed
the
jshearer/dekaf_multi_upstream_brokers
branch
from
dc394c4 to
8d847c2
Compare
jshearer
force-pushed
the
jshearer/dekaf_materialization_endpoint_support
branch
from
d1c933d to
88e456f
Compare
|
Rolling into #1840 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
As part of #1876, we want to implement broker fallback so Dekaf can connect to any of the brokers in the cluster if one doesn't respond. This adds the ability to pass a list of broker addresses, which Dekaf will try one after another until one picks up.
An improvement here would be to periodically fetch the metadata from at least one of the responding brokers and update this list of addresses so that future sessions can know about/use any newly created members of the cluster. I don't anticipate changing the topology of our cluster that frequently, and if we do then updating Dekaf's deployment configs isn't that big of a deal. I may eat my hat on this, we'll see.
In addition, we want to move people over to the new MSK cluster, so this implements routing new-style connections to a separate cluster with separate credentials.
This change is