Architecture: Sinks

A Sink is a database or cloud location made available to send Agent output to, such as a time series database like Influx or Prometheus, a document store database like Elasticsearch or Splunk, or cloud storage like S3 or Azure blob storage.

Sink Service (orb-sinks)

This microservice is responsible for:

• /sinks User facing CRUD management of Sink configurations, including validation of backend specific sink configuration
• /features/sinks User facing list of available sink backends
• Validating backend specific configuration based on the available backends
• Producing system events during sink CRUD operations for other microservices to consume
• Maintaining sink configurations in a repository DB
• Exposing gRPC service for other services e.g. enable the policy manager to verify a sink id, enable sink back in services to retrieving sink config

Sink Data Model

• id UUIDv4 (read only)
• name A name label field
• description Text field
• tags Orb tags field
• state String field indicating state of this sink: active or error (read only)
• error Text field indicating error message if status is error (read only)
• backend String field indicating the sink backend to use. Must match a backend from /features/sinks. Cannot change once created.
• config An JSON object representing backend specific configuration information
• ts_created A timestamp of date creation

Sink Feature Data Model

• backend String field of the name of the backend. Used when creating new sinks in the backend sink field
• description Text field of the description of the backend
• config A JSON object of a description the backend configuration key/values that the UI should collect.

const SETTINGS_EXAMPLE = {
  prometheus: [
    {
      type: 'text',
      input: 'text',
      title: 'Remote Host',
      name: 'remote_host',
      required: true,
    },
    {
      type: 'text',
      input: 'text',
      title: 'Username',
      name: 'username',
      required: true,
    },
    {
      type: 'password',
      input: 'text',
      title: 'Password',
      name: 'password',
      required: true,
    },
    {
      input: 'select',
      title: 'Select something',
      name: 'Select ',
      options: [
        {
          value: 'opt-1',
          label: 'Opt 1',
        },
      ],
      required: true,
    },
  ],
};

Backend Sink Handler Service [orb-sinker]

This service consumes agent backend metric output and implements different sink backend types, able to integrate with their specific database or cloud service.

This microservice is responsible for:

• Backend specific sink integration functionality: communicating with the specific backend
• Consuming appropriate metric output from the agents, handle format conversion/deserialization/decompression/etc as necessary, and sending it to the configured sink backend
• Maintaining status of the connection to the sink backend, including indicating failure state such as unable to connect
• Communicating with sink service via gRPC to pull configurations as necessary
• Consuming system events from the sink service to understand when sink configurations add, change, or delete
• Consuming system events from the policy service to understand the association between datasets that apply to this sink type, and which configured back ends to send the data to

These microservices have no user facing REST API.

Current supported sink backends include:

• prometheus

Prometheus Sink Backend

This backend translates agent metric output to Prometheus format, and sends it to a Prometheus time series databases via remote_write protocol, based on a given Sink configuration.

Required sink specific configuration data (must exist in config field):

• remote_host: The remote host to remote_write the prometheus metrics to
• username: If required by prometheus setup, the user name to connect
• password: If required by prometheus setup, the password to connect

Note: more options may be necessary, reference https://prometheus.io/docs/prometheus/2.27/configuration/configuration/#remote_write

Reference remote_write implementations:

• https://github.com/m3dbx/prometheus_remote_client_golang
• https://github.com/timescale/promscale/blob/master/docs/writing_to_promscale.md