logging

loki

https://github.com/grafana/loki

• loki.yaml

EFK stack

• elasticsearch.yaml - elasticsearch - uses a TON of memory
• fluentd.yaml - fluentd for collecting all container logs from the kubernetes cluster
• kibana.yaml - kibana log viewer UI

Elasticsearch index templates

See index templates for details. It appears that it is necessary to manually create index templatess for elasticsearch in order to apply deletion policies.

This is what I have configured for elasticsearch:

New delete-after-30-days ILM policy

PUT _ilm/policy/delete-after-30d
{
  "delete-after-30d" : {
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "30d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

Index patterns

for logstash-*:

PUT _template/logstash
{
  "index_patterns": ["logstash-*"]
}

for fluentd-syslog-*:

PUT _template/fluentd-syslog
{
  "index_patterns": ["fluentd-syslog-*"]
}

... After creating the index patterns, it is necessary to 'apply' the index patterns to the newly-created ILM deletion policy so that all new indexes created (with the above patterns) will have the policy automatically associated.

remote syslog logging

• fluentd.yaml - fluentd (deployed as fluentd-syslog) for listening as a syslog server on UDP/5140 for collecting all syslog messages from all hosts external from the cluster

  • Configure the remote rsyslog daemon with something like,

  echo "*.* @10.2.0.104:5140" > /etc/rsyslog.d/10-fluentd.conf && service rsyslog restart