"you mean apache/nginx" - bbrazil
This provides a simple reverse proxy for prometheus exporters. It is intended as a single binary alternative to nginx/apache for use in environments where opening multiple TCP ports to all servers might be difficult (technically or politically)
The advantages are:
- A single port can be used to query multiple exporters (to ease firewall configuration concerns).
- Can provide TLS with optional client certificate authentication.
- Provides verification that the target is serving prometheus metrics.
- Can be used to execute scripts that produce prometheus metrics.
- up behaviour is the same as for querying individual collectors.
- Small code size, minimal external depedencies, easily auditable.
The exporter has three endpoints.
-
/: displays a list of all exporters with links to their metrics.
- Returns JSON if the header "Accept: application/json" is passed
-
/proxy: which takes the following parameters:
- module: the name of the module from the configuration to execute.
- args (optional): arguments to pass to the module.
- params (optional): named parameter to pass to the module (either as CLI args, or http parameters).
-
/metrics: this exposes the metrics for the collector itself.
Features that will NOT be included:
- merging of module outputs into one query (this would break up behaviour)
You can build directly using a plain go get github.com/QubitProducts/exporter_exporter.
The provided Makefile is primarily used for releases.
Pre-built binaries and a debian package are available on the GitHub release page.
An ansible recipe as also available (kindly provided by one of our users).
TODO:
- Config reload on HUP (or POST, or config file change?)
- route to a docker/rocket container by name
The binary can be installed as a Windows service by supplying the -winsvc install arg.
All other arguments passed along with -winsvc install will be added to the service startup
and can only be changed by uninstalling/installing it again (or modifying the Windows registry directly).
In expexp.yaml list each exporter listening on localhost with its known port.
modules:
node:
method: http
http:
port: 9100
mtail:
method: http
http:
port: 3903
cadvisor:
verify: false
method: http
http:
port: 4194
netdata:
method: http
http:
port: 19999
path: '/api/v1/allmetrics?format=prometheus'
somescript:
method: exec
timeout: 1s
exec:
command: /tmp/myscript.sh
args:
- "myarg1"
- "myarg2"
env:
THING: "1"
THING2: "2"
In your prometheus configuration
scrape_configs:
- job_name: 'expexp_metrics'
scrape_interval: 1s
static_configs:
- targets: ['host:9999']
- job_name: 'cadvisor'
scrape_interval: 5s
metrics_path: /proxy
params:
module:
- cadvisor
static_configs:
- targets: ['host:9999']
- job_name: 'mtail'
scrape_interval: 5s
metrics_path: /proxy
params:
module:
- mtail
static_configs:
- targets: ['host:9999']
- job_name: 'somescript'
scrape_interval: 5s
metrics_path: /proxy
params:
module:
- somescript
static_configs:
- targets: ['host:9999']
You can also specify -config.dirs to break the configuration into separate
files. The module name is taken from the name of the file (minus the
yml/yaml extension), and the configuration for that module goes in at the
top level.
==> expexp.yaml <==
modules: {}
==> expexp.d/node.yaml <==
method: http
http:
port: 9100
==> expexp.d/mtail.yaml <==
method: http
http:
port: 3903
You can use exporter_exporter with TLS to encrypt the traffic, and at the same time enforce strong mutual authentication between the nodes and the prometheus server.
Note that -web.tls.verify will accept any certificate signed by the
-web.tls.ca, so you need to create a separate CA for this purpose - or use
a self-signed certificate, which acts as its own CA.
Here is a simple configuration example, using one key/cert for the prometheus server and one key/cert shared between all the remote nodes. Firstly, create the keys and certs:
openssl req -x509 -newkey rsa:1024 -keyout prom_node_key.pem -out prom_node_cert.pem -days 29220 -nodes -subj /commonName=prom_node/
openssl req -x509 -newkey rsa:1024 -keyout prometheus_key.pem -out prometheus_cert.pem -days 29220 -nodes -subj /commonName=prometheus/
Create an /etc/prometheus/ssl/ directory on the prometheus server and all
the remote nodes. Install both cert.pem files everywhere. It is safe for
them to be world-readable.
Install prom_node_key.pem only on the nodes, and set file permissions to
protect it so that only exporter_exporter can read it. Similarly, install
prometheus_key.pem only on the prometheus server, and set permissions so
that only the prometheus process can read it.
Configuration for exporter_exporter on the nodes (here it also disables plain HTTP):
EXPEXP_FLAGS='-web.listen-address= -web.tls.listen-address=:9998
-web.tls.cert=/etc/prometheus/ssl/prom_node_cert.pem
-web.tls.key=/etc/prometheus/ssl/prom_node_key.pem
-web.tls.ca=/etc/prometheus/ssl/prometheus_cert.pem
-web.tls.verify'
To test, use curl to make a scrape, replacing x.x.x.x with the IP address
of the target:
curl --cert /etc/prometheus/ssl/prometheus_cert.pem \
--key /etc/prometheus/ssl/prometheus_key.pem \
--cacert /etc/prometheus/ssl/prom_node_cert.pem \
--resolve prom_node:9998:x.x.x.x \
-v https://prom_node:9998/proxy?module=node
When this is working, configure your prometheus server to use https. Example:
- job_name: node
scrape_interval: 1m
scrape_timeout: 50s
file_sd_configs:
- files:
- /etc/prometheus/targets.d/node_targets.yml
scheme: https
tls_config:
# Verifying remote identity
ca_file: /etc/prometheus/ssl/prom_node_cert.pem
server_name: prom_node
# Asserting our identity
cert_file: /etc/prometheus/ssl/prometheus_cert.pem
key_file: /etc/prometheus/ssl/prometheus_key.pem
metrics_path: /proxy
params:
module: [ node ]
relabel_configs:
- source_labels: [__address__]
target_label: instance
- source_labels: [__address__]
regex: '[^:]+'
target_label: __address__
replacement: '${1}:9998'
Example /etc/prometheus/targets.d/node_targets.yml:
- labels: []
targets:
- 192.0.2.1
- 192.0.2.2