Architecture: Sinks

A Sink is a database or cloud location made available to send Agent output to, such as a time series database like Influx or Prometheus, a document store database like Elasticsearch or Splunk, or cloud storage like S3 or Azure blob storage.

Sink Service (orb-sinks)

This microservices responsible for:

• /sinks User facing CRUD management of Sink configurations, including validation of backend specific sink configuration
• /features/sinks User facing list of available sink backends
• Producing system events during sink CRUD operations for other microservices to consume
• Maintaining sink configurations in a repository DB

Data Model

• id UUIDv4 (read only)
• name A name label
• description Text field
• tags Orb tags field
• status String field indicating status of this sink: active or error (read only)
• error Text field indicating error message if status is error (read only)
• backend String field indicating the sink backend to use. Must match a backend from /features/sinks. Cannot change once created.
• config A schemaless JSON object representing backend specific configuration information

Backend Sink Handler Services

These services represent the implementation of different sink types, able to integrate with their specific database or cloud service.

These microservices are responsible for:

• Backend specific sink integration functionality: communicating with the specific backend
• Consuming appropriate metric output from the agents, handle deserialization/decompression if necessary, and sending it to the configured backend
• Maintaining status of the connection to the backend, including indicating failure state such as unable to connect
• Communicating with sink service via gRPC to pull configurations as necessary
• Consuming system events from the sink service to understand when sink configurations add, change, or delete
• Consuming system events from the policy service to understand the association between policies that apply to this sink type, and which configured back ends to send the data to

These microservices have no user facing REST API.

Current supported sink backends include:

• prometheus

Prometheus Sink (orb-prom-sink)

This microservices implements the integration to Prometheus time series databases using remote_write

Required sink specific configuration data:

• remote_host: The remote host to remote_write the prometheus metrics to
• username: If required by prometheus setup, the user name to connect
• password: If required by prometheus setup, the password to connect

Note: more options may be necessary, reference https://prometheus.io/docs/prometheus/2.27/configuration/configuration/#remote_write