Docker image based on official fluentd, installing necessary plugins for use with AWS ES
The container is supposed to listen on port 24224 (or FLUENTD_PORT). Since docker has a fluentd log driver, it can get used to ingest other container's log messages.
The messages can get forwarded to an AWS ES instance via https transport.
One drawback of the fluentd log driver is, that the messages are not locally available with docker logs. Therefore, 3 example configs are included to ship logs to 1) AWS ES, 2) a mountable local file, 3) both 1 and 2.
There are 2 ways to provide the config:
This takes one of the existing or any other self-created config template, replaces some env variables and copies it to fluent.conf.
This file is picked up in the docker-build-target and copied into the image and is the preferred way to prepare an image for deployment.
Example:
make prepare-config \
AWS_ELASTICSEARCH_URL=https://search-awslogs-xxx.eu-west-1.es.amazonaws.com \
AWS_REGION=eu-west-1 \
CONF_FILE_TEMPLATE=fluent-aws.conf.tmplAlternatively, the config can be mounted into the container at runtime. This can be done using an environment variable for the respective make target:
make docker-run CONF_FILE=$(pwd)/fluent.confThis is the preferred way for local/testing scenarios.
| Variable Name | Default | Description |
|---|---|---|
| STAGE | (empty) | When set, will set the container-name accordingly. make docker-run STAGE=dev => container name: fluentd-aws-elasticsearch-dev |
| FLUENTD_PORT | 24224 | Which port to export from the container to listen on for incoming log messages. |
| AWS_REGION | eu-west-1 | Used in substitution in case of target prepare-config |
| AWS_ELASTICSEARCH_URL | (empty) | Used in substitution in case of target prepare-config |
| CONF_FILE_TEMPLATE | (empty) | Used as base config to apply substitutions to in case of target prepare-config |
| CONF_FILE | (empty) | abs path to local conf file which should get mounted into the container with target docker-run |
The base setup assumes that the container is allowed to ship messages to AWS ES by role or IP. In case there is a need to fall back to AWS key and secret, this can be done by amending the config file.