Logstash AWS Kinesis Input Plugin

This is a AWS Kinesis input plugin for Logstash. Under the hood uses the Kinesis Client Library.

Installation

This plugin requires Logstash >= 1.5, and can be installed by Logstash itself.

$(LOGSTASH_DIR)/bin/plugin install logstash-input-kinesis

input {
  kinesis {
    kinesis_stream_name => "my-logging-stream"
    codec => json { }
  }
}

This are the properties you can configure and what are the default values:

application_name: The name of the application used in DynamoDB for coordination. Only one worker per unique stream partition and application will be actively consuming messages.
- required: false
- default value: logstash
kinesis_stream_name: The Kinesis stream name.
- required: true
region: The AWS region name for Kinesis, DynamoDB and Cloudwatch (if enabled)
- required: false
- default value: us-east-1
checkpoint_interval_seconds: How many seconds between worker checkpoints to DynamoDB. A low value ussually means lower message replay in case of node failure/restart but it increases CPU+network ussage (which increases the AWS costs).
- required: false
- default value: 60
metrics: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library.
- required: false
- default value: nil

This plugin uses the default AWS SDK auth chain, DefaultAWSCredentialsProviderChain, to determine which credentials the client will use.

The default chain follows this order trying to read the credentials:

The credentials will need access to the following services:

AWS Kinesis
AWS DynamoDB: the client library stores information for worker coordination in DynamoDB (offsets and active worker per partition)
AWS CloudWatch: if the metrics are enabled the credentials need CloudWatch update permisions granted.

Look at the documentation for deeper information.