Kafka Connector

Contents

• Overview
• Summary
• Source Files
• Configuration
  • Shared Configuration
  • Connector-Specific Configuration
• Diagnostics
• Metrics

Overview

KafkaConnector is intended for consuming a single Kafka topic

Summary

Supports Datastream Updates	Y
Checkpoint Type	Kafka consumer offsets are regularly committed to Kafka (see commitIntervalMs)
Diagnostics Aware	Y

Source Files

1. com.linkedin.datastream.connectors.kafka.KafkaConnector

2. com.linkedin.datastream.connectors.kafka.AbstractKafkaConnector

Configuration

• All connector config properties must be prefixed with brooklin.server.connector.<connectorName>.

• connectorName is an arbitrary user-supplied name specified in Brooklin configuration (brooklin.server.connectorNames)

Shared Configuration

Configuration properties shared among all Kafka connectors in Brooklin are documented on this page.

Connector-Specific Configuration

Configuration properties specific to KafkaConnector are listed below.

Property	Description	Default
isGroupIdHashingEnabled	A flag indicating whether Kafka consumer group ID should be hashed If true, the consumer group ID for a datastream is set to <clusterName>.<groupIdMD5Hash>, where: <clusterName>: Brooklin cluster name <groupIdMD5Hash>: The MD5 hash of metadata.system.taskPrefix property of the datastream If false, the consumer group ID for a datastream is set to <sourceConnectionString>-to-<destConnectionString>, where: <sourceConnectionString>: source.connectionString of the datastream <destConnectionString>: destination.connectionString of the datastream	false
whiteListedClusters	A comma-separated list of Kafka brokers, e.g. host1:port1,host2:port2 Any datastream using this connector must specify one of these Kafka brokers in its source.connectionString	(None)

Diagnostics

URL

GET /diag?q=:query&type=:component&scope=:scope&content=:componentQuery

URL Params

Required query Possible values are: status or allStatus status retrieves data for a single Brooklin instance allStatus retrieves aggregated data for all Brooklin instances in the cluster type: only supported value is connector scope: Name of the connector to query, as specified in brooklin.server.connectorNames content Represents a subquery to the component (connector) in question Possible values are: datastream_state and position Subquery datastream_state Subquery Params Required datastreamName: Name of datastream to query Example datastream_state?datastream=:datastreamName Constraints Datastream must exist Response Content-Type application/json Schema A paged list of one or more ServerComponentHealth objects, each with its status property set to a JSON-serialized KafkaDatastreamStatesResponse The ServerComponentHealth objects appear under an elements key as in the example below { "elements": [ ], "paging": { "count": "[int]", "start": "[int]", "links": "[array]" } } Subquery position Subquery Params Optional offsets A boolean indicating whether to return time-based vs. offset-based position data for the retrieved DatastreamTasks Default is false (time-based) Example position?offsets=true Constraints Setting it to true requires enableKafkaPositionTracker to be true datastream: Name of datastream to limit results to Example position?datastream=:datastreamName Constraints Datastream must exist topic: Name of Kafka topic to limit results to Example position?topic=:topicName Constraints Topic must exist Response Content-Type application/json Schema A paged list of one or more ServerComponentHealth objects, each with its status property set to a JSON-serialized DatastreamPositionResponse The ServerComponentHealth objects appear under an elements key as in the example below { "elements": [ ], "paging": { "count": "[int]", "start": "[int]", "links": "[array]" } }

Metrics

All Kafka connectors share the common metrics documented at Kafka Connectors Shared Configuration and Data Models.