Streaming System Design Doc

This document describes how observations from the Array of Things (and, hopefully, other sensor networks, eventually) get into Plenario and out to users.

Contents

1. Overview
2. Beehive
3. Publish Lambda
4. Resolve Lambda
5. Socket.io Server
6. Apiary

1: Beehive

How it works:

Beehive communicates directly with nodes out in the wild. It lives off-cloud, so we need some interface to get sensor observations from Beehive into Plenario. We want a mailbox that has minimal downtime, and that a consuming application can pull records off of at its own pace. Kinesis fits the bill. Beehive pushes observations one by one using boto to a Kinesis Stream that the Plenario team provisions - the "Publication Stream".

Scaling concerns:

We have created a one-off access key that enables Beehive to push to that stream and given it to the Waggle maintainer. For now there is just one beehive, living at Argonne. To enable multiple beehives and non-Waggle sensor networks to publish observations, we will likely need to automate the credential-making process. Once we have multiple network maintainers pushing to the same stream, we'll also want to authenticate the sender.

2: Publish Lambda

Lambda function that reads observations from Kinesis and sends them down two paths: into Firehose to save in a database and to Redis for streaming servers to publish.

3: Resolve Lambda

Lambda function that batch updates t

4: Socket.io Server

5: Apiary (Metadata Management)