diff --git a/README.md b/README.md index f7db5e2..c8ec28c 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,9 @@ # Logstash AWS Kinesis Input Plugin -This is a plugin for [Logstash](https://github.com/elasticsearch/logstash). - [![Build Status](https://travis-ci.org/codekitchen/logstash-input-kinesis.svg)](https://travis-ci.org/codekitchen/logstash-input-kinesis) +This is a [AWS Kinesis](http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html) input plugin for [Logstash](https://github.com/elasticsearch/logstash). Under the hood uses the [Kinesis Client Library](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html). + ## Installation This plugin requires Logstash 1.5, and can be installed by Logstash @@ -24,7 +24,40 @@ input { } ``` -At least `kinesis_stream_name` is requred. +## Configuration + +This are the properties you can configure and what are the default values: + +* `application_name`: The name of the application used in DynamoDB for coordination. Only one worker per unique stream partition and application will be actively consuming messages. + * **required**: false + * **default value**: `logstash` +* `kinesis_stream_name`: The Kinesis stream name. + * **required**: true +* `region`: The AWS region name for Kinesis, DynamoDB and Cloudwatch (if enabled) + * **required**: false + * **default value**: `us-east-1` +* `checkpoint_interval_seconds`: How many seconds between worker checkpoints to DynamoDB. A low value ussually means lower message replay in case of node failure/restart but it increases CPU+network ussage (which increases the AWS costs). + * **required**: false + * **default value**: `60` +* `metrics`: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library. + * **required**: false + * **default value**: `nil` + +## Authentication + +This plugin uses the default AWS SDK auth chain, [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html), to determine which credentials the client will use. + +The default chain follows this order trying to read the credentials: + * `AWS_ACCESS_KEY_ID` / `AWS_SECRET_KEY` environment variables + * `~/.aws/credentials` credentials file + * EC2 instance profile + +The credentials will need access to the following services: +* AWS Kinesis +* AWS DynamoDB: the client library stores information for worker coordination in DynamoDB (offsets and active worker per partition) +* AWS CloudWatch: if the metrics are enabled the credentials need CloudWatch update permisions granted. + +Look at the [documentation](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html) for deeper information. ## Contributing