About

kinesis-client-node is a node.js kinesis client library that interfaces with Amazon's MultiLangDaemon

License

MIT license

Support

• No warranty or support is implied.
• No representations made towards quality or performance.
• Feel free to raise issues or fork.
• Feedback / Suggestions / Patches welcome.

Prerequisites

1. Install Java SDK
2. Setup the AWS command line utlilities
3. Create an IAM role that has access to Kinesis, CloudWatch and DynamoDB
4. Create a credentials file - ~/.aws/credentials
5. Add a [default] profile with aws_access_key_id and aws_secret_access_key of the IAM role created in step 3

Installing Dependencies

Step 1

Create a Javascript file named kinesis.app.words.settings.js under coffee/sample/

Add the following settings (replace with real values)

settings = {
	"aws" : {
		"region"  : "AWS Region",
		"accessKeyId" : "AWS Access Key",
		"secretAccessKey" : "AWS Secret Access Key"
	},
	"app" : {
		"recordProcessor" : {
			"streamName" : "Name of Kinesis Stream",
			"sleepSeconds" : "Sleep Interval (integer)",
			"checkpointRetries" : "Number of checkpoint retries (integer)",
			"checkpointFreqSeconds" : "Checkpoint frequency in seconds (integer)"
		}
	}
}

module.exports = settings

Step 2

Install package dependencies

npm install

Step 3

Install gulp globally

npm install -g gulp

Step 4

Run the setup task to fetch KCL jars

gulp setup

Running the sample application

The sample app is a word processor composed of

• A word producer that puts words into a Kinesis stream - kinesis.producer.words.coffee
• A record processor that handles initialialisation, shutdown and record processing - recordprocessor.coffee
• The sample app code that is handed each record recieved by the record processor for processing - kinesis.app.words.coffee

Start the kinesis daemon

Generate the Java command to start the daemon using

./lib/kclhelper.js --print_command --java /usr/bin/java --props ./lib/sample/ml-daemon.properties

Execute the generated command

Start the word processor

The word producer can be invoked by running

./lib/sample/kinesis.producer.words.js

It accepts the following arguments

--stream STREAM_NAME
--region AWS_REGION 
--words  COMMA_SEP_WORD_LIST
--period PUT_REPEATEDLY_AFTER_MILLIS

Example

./lib/sample/kinesis.producer.words.js --stream words --region ap-southeast-2 --words one,two,three --period 5000

Tail Logs

Tail the log file - ./logs/worker.log to see INFO messages generated by KCL

Todo

• Testcases