Kafka Connector

Contents

• Overview
• Summary
• Source Files
• Configuration
  • Shared Configuration
  • Connector-Specific Configuration
• Diagnostics
• Metrics

Overview

KafkaConnector is intended for consuming a single Kafka topic

Summary

Supports Datastream Updates	Y
Checkpoint Type	Kafka consumer offsets are regularly committed to Kafka (see commitIntervalMs)
Diagnostics Aware	Y

Source Files

1. com.linkedin.datastream.connectors.kafka.KafkaConnector

2. com.linkedin.datastream.connectors.kafka.AbstractKafkaConnector

Configuration

• All connector config properties must be prefixed with brooklin.server.connector.<connectorName>.

• connectorName is an arbitrary user-supplied name specified in Brooklin configuration (brooklin.server.connectorNames)

Shared Configuration

Configuration properties shared among all Kafka connectors in Brooklin are documented on this page.

Connector-Specific Configuration

Configuration properties specific to KafkaConnector are listed below.

Property	Description	Default
isGroupIdHashingEnabled	A flag indicating whether Kafka consumer group ID should be hashed If true, the consumer group ID for a datastream is set to <clusterName>.<groupIdMD5Hash>, where: <clusterName>: Brooklin cluster name <groupIdMD5Hash>: The MD5 hash of metadata.system.taskPrefix property of the datastream If false, the consumer group ID for a datastream is set to <sourceConnectionString>-to-<destConnectionString>, where: <sourceConnectionString>: source.connectionString of the datastream <destConnectionString>: destination.connectionString of the datastream	false
whiteListedClusters	A comma-separated list of Kafka brokers, e.g. host1:port1,host2:port2 Any datastream using this connector must specify one of these Kafka brokers in its source.connectionString	(None)

Diagnostics

All Kafka connectors share the common diagnostic endpoints documented at Kafka Connectors Shared Configuration and Data Models.

Metrics

All Kafka connectors share the common metrics documented at Kafka Connectors Shared Configuration and Data Models.