Kafka Connector

Contents

• Overview
• Summary
• Source Files
• Configuration
  • Shared Configuration
  • Connector-Specific Configuration
• Diagnostics
• Metrics

Overview

• KafkaConnector is intended for consuming from a single Kafka topic.

• For Kafka mirroring use cases, please refer to KafkaMirrorMakerConnector instead.

Summary

Supports Datastream Updates	Y
Checkpoint Type	Kafka consumer offsets are regularly committed to Kafka (see commitIntervalMs)

Source Files

1. com.linkedin.datastream.connectors.kafka.KafkaConnector

2. com.linkedin.datastream.connectors.kafka.AbstractKafkaConnector

Configuration

• All connector config properties must be prefixed with brooklin.server.connector.<connectorName>.

• connectorName is an arbitrary user-supplied name specified in Brooklin configuration (brooklin.server.connectorNames)

Shared Configuration

Configuration properties shared among all Kafka connectors in Brooklin are documented in Kafka Connectors Shared Logic.

Connector-Specific Configuration

Configuration properties specific to KafkaConnector are listed below.

Property	Description	Default
whiteListedClusters	A comma-separated list of Kafka brokers, e.g. host1:port1,host2:port2 Any datastream using this connector must specify one of these Kafka brokers in its source.connectionString	(None)

Diagnostics

All Kafka connectors share the common diagnostic endpoints documented at Kafka Connectors Shared Logic.

Metrics

All Kafka connectors share the common metrics documented at Kafka Connectors Shared Logic.