-
Notifications
You must be signed in to change notification settings - Fork 137
Kafka MirrorMaker Connector
-
KafkaMirrorMakerConnector
is intended for consuming multiple topics in a Kafka cluster via regular expression patterns. -
KafkaMirrorMakerConnector
is optimized for Kafka clusters mirroring scenarios.
Supports Datastream Updates |
Yes |
---|---|
Checkpoint Type |
Kafka consumer offsets are regularly committed to Kafka (see |
Since KafkaMirrorMakerConnector
consumes data from the the source Kafka cluster using a Kafka consumer, it needs to periodically commit the offsets it retrieved and processed. To guarantee it is actually safe to do so, it needs to flush all the data it consumed to the DatastreamEventProducer
first. In Kafka mirroring scenarios, this would cause the transport provider producing data to the destination Kafka cluster (i.e. KafkaTransportProvider
) to flush the Kafka producer(s) it encapsulates — a synchronous operation that may have adverse effects on throughput in high load scenarios.
To address this concern, KafkaMirrorMakerConnector
features an optional flushless produce mode, where it avoids flushing the DatastreamEventProducer
altogether, and relies on the asynchronous ack notifications it receives from the DatastreamEventProducer
to commit the offsets that have been delivered only.
-
All connector config properties must be prefixed with
brooklin.server.connector.<connectorName>.
-
connectorName
is an arbitrary user-supplied name specified in Brooklin configuration (brooklin.server.connectorNames
)
-
Configuration properties shared among all Kafka Connectors in Brooklin are documented in Kafka Connectors Shared Logic.
-
This connector has the following defaults for the configuration property(ies) listed below.
Property Description Default consumer.*
Kafka consumer configuration properties
-
enable.auto.commit
:false
-
auto.offset.reset
:earliest
-
Configuration properties specific to KafkaMirrorMakerConnector
are listed below.
Property | Description | Default |
---|---|---|
Indicates whether Flushless Produce mode is enabled |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(None) |
Property | Description | Default |
---|---|---|
|
|
|
All Kafka connectors share the common diagnostic endpoints documented at Kafka Connectors Shared Logic.
All Kafka connectors share the common metrics documented at Kafka Connectors Shared Logic.
- Home
- Brooklin Architecture
- Production Use Cases
- Developer Guide
- Documentation
- REST Endpoints
- Connectors
- Transport Providers
- Brooklin Configuration
- Test Driving Brooklin