Architecture: Sinks

A Sink is a database or cloud location made available to send Agent output to, such as a time series database like Influx or Prometheus, a document store database like Elasticsearch or Splunk, or cloud storage like S3 or Azure blob storage.

Sink Service (orb-sinks)

This microservice is responsible for:

• /sinks User facing CRUD management of Sink configurations, including validation of backend specific sink configuration
• /features/sinks User facing list of available sink backends
• Producing system events during sink CRUD operations for other microservices to consume
• Maintaining sink configurations in a repository DB
• Exposing gRPC service for other services e.g. enable the policy manager to verify a sink id, enable sink back in services to retrieving sink config
• Validating backend specific configuration based on the available backends

Sink Data Model

• id UUIDv4 (read only)
• name A name label field
• description Text field
• tags Orb tags field
• status String field indicating status of this sink: active or error (read only)
• error Text field indicating error message if status is error (read only)
• backend String field indicating the sink backend to use. Must match a backend from /features/sinks. Cannot change once created.
• config An JSON object representing backend specific configuration information

Sink Feature Data Model

• backend String field of the name of the backend. Used when creating new sinks in the backend sink field
• description Text field of the description of the backend

Backend Sink Handler Services

These services represent the implementation of different sink types, able to integrate with their specific database or cloud service.

These microservices are responsible for:

• Backend specific sink integration functionality: communicating with the specific backend
• Consuming appropriate metric output from the agents, handle deserialization/decompression if necessary, and sending it to the configured backend
• Maintaining status of the connection to the backend, including indicating failure state such as unable to connect
• Communicating with sink service via gRPC to pull configurations as necessary
• Consuming system events from the sink service to understand when sink configurations add, change, or delete
• Consuming system events from the policy service to understand the association between policies that apply to this sink type, and which configured back ends to send the data to

These microservices have no user facing REST API.

Current supported sink backends include:

• prometheus

Prometheus Sink (orb-prom-sink)

This microservices implements the integration to Prometheus time series databases using remote_write

Required sink specific configuration data (must exist in config field):

• remote_host: The remote host to remote_write the prometheus metrics to
• username: If required by prometheus setup, the user name to connect
• password: If required by prometheus setup, the password to connect

Note: more options may be necessary, reference https://prometheus.io/docs/prometheus/2.27/configuration/configuration/#remote_write