architecture.md

Table of Contents generated with DocToc

• Coordinator
  • Sharding
• Core
  • Ingestion Flow
• Memory
• Cassandra
• Kafka
• Spark
• HTTP
• Standalone
• Gateway
• CLI

The code is laid out following the different parts and components.

Coordinator

The Coordinator provides an upper-level, client-facing interface for a FiloDB cluster and its core per-node components. It also handles shard management, assignment, and status handling for ingestion and distributed in-memory querying through an internal PubSub CQRS component. This is composed of

• A configurable ShardAssignmentStrategy, responsible for assigning or removing shards to/from nodes based a defined policy, when state changes occur. It is possible to locate the shard and node coordinator to query with either a partition hash or a shard key hash and # bits.
• ShardMapper(s) which keep track of the mapping between shards and nodes for a single dataset. For the cluster these are managed by the ShardCoordinatorActor, a child of the cluster singleton. On each node, the NodeClusterActor tracks them locally.

The coordinator also handles different types of streaming data ingestion, and the ability to subscribe to ingestion stream shard health, state and status.

Each FiloDB node creates a node guardian which creates and supervises the lifecycle of the primary components: NodeCoordinatorActor, local proxy to the cluster singleton NodeClusterActor, node metrics, and handles graceful shutdown of the node.

The NodeCoordinatorActor is the common external API entry point for all FiloDB operations. For each new dataset added the NodeCoordinatorActor creates two actors for the new dataset

• An IngestionActor which creates and manages sharded ingestion streams of records for that dataset, and stores the records in the memstore
• A QueryActor which translates external query API calls into internal ColumnStore calls. The actual reading of data structures and aggregation is performed asynchronously by Observables

The IngestionActor publishes shard ingestion change events to the ShardCoordinatorActor which then broadcasts to all subscribers of that dataset. The QueryActor is decoupled from the ShardCoordinatorActor and is subscribed by its parent, the NodeCoordinatorActor, to these shard change events and simply receives them and can act on them as needed.

Sharding

• Shard Coordination
• Shard Assignment
• Shard Event Subscriptions
• Subscribe To Shard Status And Ingestion Events

Core

These components form the core part of FiloDB and are portable across data stores. Subcomponents:

• binaryrecord - used for supporting efficient, no-serialization, multi-schema partition keys and ingestion records
• memstore - a MemStore ingests records, encodes them into columnar chunks, and allows for real-time querying through the ChunkSource API. The current implementation is a TimeSeriesMemStore which is designed for very high cardinality time series data. For each dataset it stores one or more shards, each of which may contain many many thousands of TimeSeriesPartition instances. MemStores also manage persistence of encoded data via ChunkSinks, as well as read-through caching of in-memory chunks from a persistent chunk store.
• store - contains the main APIs for persistence and chunk reading, including ChunkSource and ChunkSink, as well as the MetaStore for metadata persistence. Most of the APIs are based on reactive streams for backpressure handling.
• query - contains aggregation and querying logic built on top of ChunkSources.
• metadata - Dataset and Column definitions

FiloDB's Dataset defines the partition and data columns and the row key. The MetaStore defines an API for concurrent reads/writes/updates on dataset metadata. Each Column has a ColumnType, which has a KeyType. KeyType is a fundamental type class defining serialization and extraction for each type of column/key.

Ingestion Flow

Also see the ingeston doc for details on recovery and persistence.

Memory

Contains off-heap memory management as well as columnar encoding/decoding/super fast binary vector logic.

Also contains OffheapLFSortedIDMap, an efficient, lock-free, sorted offheap map data structure used to help track our chunks and ensure minimal heap usage.

Cassandra

An implementation of ColumnStore and MetaStore for Apache Cassandra.

Kafka

Ingestion for FiloDB from Kafka event streams.

Spark

Contains the Spark input source for ingesting and querying data from FiloDB.

HTTP

Contains the FiloDB HTTP API routes

Standalone

The standalone module is used for FiloDB real-time direct ingestion (from Kafka) and low-latency querying without Spark, for metrics and event use cases.

Gateway

Gateway module contains logic to accept input metrics/data using different formats such as Influx Line Protocol, Graphite, etc. and forward it to Kafka/rest of FiloDB, as well as to generate time series data for testing.

CLI

Contains the client CLI for setting up datasets and connecting with a FiloDB standalone cluster for direct querying using PromQL.