Skip to content

Commit

Permalink
Merge pull request codekitchen#3 from samuelgmartinez/doc-improvements
Browse files Browse the repository at this point in the history
README.md doc improvements
  • Loading branch information
codekitchen committed Sep 24, 2015
2 parents c5ebf3c + f1c3f24 commit d51de84
Showing 1 changed file with 36 additions and 3 deletions.
39 changes: 36 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Logstash AWS Kinesis Input Plugin

This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).

[![Build Status](https://travis-ci.org/codekitchen/logstash-input-kinesis.svg)](https://travis-ci.org/codekitchen/logstash-input-kinesis)

This is a [AWS Kinesis](http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html) input plugin for [Logstash](https://github.com/elasticsearch/logstash). Under the hood uses the [Kinesis Client Library](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html).

## Installation

This plugin requires Logstash 1.5, and can be installed by Logstash
Expand All @@ -24,7 +24,40 @@ input {
}
```

At least `kinesis_stream_name` is requred.
## Configuration

This are the properties you can configure and what are the default values:

* `application_name`: The name of the application used in DynamoDB for coordination. Only one worker per unique stream partition and application will be actively consuming messages.
* **required**: false
* **default value**: `logstash`
* `kinesis_stream_name`: The Kinesis stream name.
* **required**: true
* `region`: The AWS region name for Kinesis, DynamoDB and Cloudwatch (if enabled)
* **required**: false
* **default value**: `us-east-1`
* `checkpoint_interval_seconds`: How many seconds between worker checkpoints to DynamoDB. A low value ussually means lower message replay in case of node failure/restart but it increases CPU+network ussage (which increases the AWS costs).
* **required**: false
* **default value**: `60`
* `metrics`: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library.
* **required**: false
* **default value**: `nil`

## Authentication

This plugin uses the default AWS SDK auth chain, [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html), to determine which credentials the client will use.

The default chain follows this order trying to read the credentials:
* `AWS_ACCESS_KEY_ID` / `AWS_SECRET_KEY` environment variables
* `~/.aws/credentials` credentials file
* EC2 instance profile

The credentials will need access to the following services:
* AWS Kinesis
* AWS DynamoDB: the client library stores information for worker coordination in DynamoDB (offsets and active worker per partition)
* AWS CloudWatch: if the metrics are enabled the credentials need CloudWatch update permisions granted.

Look at the [documentation](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html) for deeper information.

## Contributing

Expand Down

0 comments on commit d51de84

Please sign in to comment.