Skip to content

LogStash and ElasticSearch for Logging Administration Guide

Jump to bottom
jrmccombs edited this page Nov 17, 2020 · 39 revisions

Installation of LogStash Packages

This section describes how to install LogStash on VMs where log data is to be exported to the ElasticSearch service. The version of LogStash we used for CADRE is 6.x. Version 7 and above did not work with AWS ElasticSearch at the time of installation. These instructions were taken from https://www.elastic.co/guide/en/logstash/7.6/installing-logstash.html#_apt

Installing on RedHat

The following steps must be taken to install on RedHat.

  1. Install the public signing key: 
    rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
  2. Add the following to /etc/yum.repos.d in a file with .repo prefix: 
    [logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
  3. Install LogStash with: 
    sudo yum install logstash
  4. Install the Amazon ES logstash plugin with: 
    sudo -E /usr/share/logstash/bin/logstash-plugin install logstash-output-amazon_es

Installing on Ubuntu

  1. Install java if it is not already installed. You can type ‘java’ to see list of possible packages to install using apt.
  2. Download and install the Public Signing Key: 
    wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –
  3. Install the apt-transport-https package: 
    sudo apt-get install apt-transport-https
  4. Save the repository definition to /etc/apt/sources.list.d/elastic-6.x.list: 
    echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
  5. Run sudo apt-get update and the repository is ready for use. You can install it with: 
    sudo apt-get update && sudo apt-get install logstash
  6. Install the Amazon ES logstash plugin with: 
    sudo -E /usr/share/logstash/bin/logstash-plugin install logstash-output-amazon_es

Configuration of LogStash Daemon

The following sections describe how to configure the LogStash account and daemon.

  1. Add the logstash account to the adm group to enable access: 
    sudo usermod -a -G adm logstash
  2. Edit the /etc/passwd file to enable /bin/bash as the shell for the logstash account
  3. Change the ownership of the /etc/logstash directory: 
    sudo chown -R logstash /etc/logstash
  4. Assume the logstash account and change the directory to /etc/logstash: 
    sudo su – logstash
cd /etc/logstash
  5. Edit the /etc/logstash/jvm.options file and change the -Xms and -Xmx parameters to: 
    -Xms200m
-Xmx500m
  6. Edit the /etc/logstash/logstash.yml file and set the pipeline.workers to a value not greater than the number of cores and sufficient to keep up with log activity.
  7. Add the following shell variables to /etc/default/logstash on Ubuntu or /etc/sysconfig/logstash on RedHat:
    • LOGSTASH_AWS_ES_HOST – The AWS ElasticSearch host
    • LOGSTASH_AWS_ES_REGION – The region of the host
    • AWS_ACCESS_KEY_ID – Access key of account with AWS ES write permissions
    • AWS_SECRET_ACCESS_KEY – Secret key of account with AWS ES write permissions
  8. Prepare the rsyslog.conf for proper syslog configuration
    • Enable the immark module in the rsyslog.conf file under Ubuntu
      1. Under Ubuntu just uncomment the line module(load=”immark”)
    • Under RedHat configure rsyslog as follows
      1. Change the group ownership of /var/log/messages to adm
      2. Add module(load=”immark”) to the modules section of the rsyslog.conf file
      3. Add $FileCreateMode 0600 to the beginning of the rules section
      4. Add before the /var/log/messages configuration 
        $FileCreateMode 0640
$FileGroup adm
      5. Add after the /var/log/message configuration 
        1. $FileGroup root
2. $FileCreateMode 0600
  9. Edit or create the /etc/logstash/logstash-all.conf file and configure logstash for the needed log file coverage. You can follow existing configurations on CADRE systems as an example. The following is helpful documentation:
  10. For each log file to be covered by LogStash, create the corresponding index using the 'create-index.py' script. The index name should be of the form '-1, where the "-1" suffix is needed to facilitate index rollovers; in the script, set the path variable accordingly.
  11. Create a writable alias alias-<index_name> using the create-alias.py which creates the alias and maps it to the index <index-name>-1. The command with JSON payload is of the form: 
    POST https://<aws-es-instance-name>/_aliases
{
  "actions": [
    {
      "add": {
         "index": "<index-name>-1",
         "alias": "alias-<index_name>",
         "is_write_index": "true"
      }
    }
  ]
}

Configuring ElasticSearch After Adding a New Index

Policies for managing the lifecycle of the indices, heartbeat alerts of the CADRE systems being logged, and visualizations must be configured when support for new log sources is added to the ElasticSearch instance.

Current Log File Coverage with LogStash

The log files currently being covered by LogStash on various CADRE systems are given in the table below. Each log file and the system it is on is listed along with the alias to the ElasticSearch index that the log data is written to.

CADRE system Log file Description Index Alias
WoS Dataset /var/log/messages System log alias-wos-dataset-syslog
WoS Dataset /pgdata/11/logs/postgresql.csv PostgreSQL log alias-wos-dataset-postgres
WoS Dataset /n4jdata/3/logs/debug.log Neo4j debug error log alias-wos-dataset-postgres
Web Gateway /var/log/syslog System log alias-web-gateway-syslog
Web Gateway /var/log/apache2/access.log Apache access log alias-web-gateway-access
Web Gateway /var/log/apache2/error.log Apache error log alias-web-gateway-error
Janus Server /var/log/syslog Janus server system log alias-janus-server-syslog
Janus Server /home/ubuntu/janus-job-listener/cadre-janus-job-listener.log Janus server job listener log alias-janus-server-listener
Janus Server /home/ubuntu/janusgraph/logs/gremlin-server.log Janus server Gremlin log alias-janus-server-gremlin
CADRE Listener /var/log/syslog CADRE listener system log alias-cadre-listener-syslog
CADRE Listener /home/ubuntu/cadre-job-listener/cadre_job_listener.log CADRE job listener log alias-cadre-job-listener
CADRE Listener /home/ubuntu/cadre-job-listener/cadre_package_listener.log CADRE package listener log alias-cadre-package-listener
CADRE Listener /home/ubuntu/cadre-job-listener/cadre_tool_listener.log CADRE tool listener log alias-cadre-tool-listener
CADRE Interface /var/log/httpd/access_log CADRE interface Apache access log alias-cadre-interface-access
CADRE Interface /var/log/httpd/error_log CADRE interface Apache error log alias-cadre-interface-error
CADRE Login /home/ubuntu/cadre-login/cadre_logging.log CADRE login log alias-cadre-login

Adding policy support for a new index

After LogStash has been configured on a VM to forward data to a new index, ElasticSearch policies must be configured to manage the new index. These policies are used mainly to control rollovers of indices once they reach a certain age, allowing the ElasticSearch system to age out old data automatically. The policies that currently exist to support CADRE indices are described in the table below. The policies are implemented under the OpenDistro policy system which defines a finite state machine for each policy. The CADRE policies support two states, a hot state for indices actively being written two through their corresponding aliases and a warm state for older indices that are retained for analysis. After a certain period of time, indices in the warm state are deleted. The policies should be created according to the instructions at https://opendistro.github.io/for-elasticsearch-docs/docs/ism/

Policy Name Description Index age to transition from Hot state Index age to delete from Warm state
syslog_policy Manages indices for Linux syslog logs 7 days 60 days
wos_database_policy Manages indices for Web of Science PostgreSQL and Neo4j database logs 7 days 90 days
web_gateway_error_policy Manages indices for web gateway Apache error logs 7 days 90 days
web_gateway_access_policy Manages indices for web gateway Apache access logs 7 days 30 days
janus_server_startup_policy Manages indices for Janus server front end startup logs 7 days 90 days
janus_server_gremlin_policy Manages indices for Janus server Gremlin logs 7 days 90 days
janus_listener_policy Manages indices for Janus server front end listener logs 7 days 90 days
cadre_listener_policy Manages indices for CADRE Jupyter listener logs 7 days 90 days
cadre_interface_access_policy Manages indices for CADRE interface Apache access logs 7 days 30 days
cadre_interface_error_policy Manages indices for CADRE interface Apache error logs 7 days 90 days
cadre_login_policy Manages indices for CADRE login error logs 7 days 90 days

When an index is rolled over by the policy a new index is created under the index name with the integer suffix incremented. However, the same policy will not be applied to the new index unless an index template is created and applied to indices newly created during index rollovers. One template is created for each log that is to be covered by LogStash. The template should be applied to the initial index (<index-name>-1) and include the policy to be applied and the rollover alias. To create the new policy template, the create-template.py script should be configured and executed once the index alias and policy have been created. The command the script issues is of the form:

POST “https://<aws-es-instance-name>/_template/<index-name>_template”
{
  "index_patterns": [
    "<index-name>*"
  ],
  "settings": {
    "opendistro.index_state_management.policy_id":          
       "<index-name>_policy",
    "opendistro.index_state_management.rollover_alias":
       "alias-<index-name>"
  }
}

The index policies can be viewed under the Index Management panel in Kibana. After the index template is successfully applied, the corresponding policy should be applied using the Index Management / Indices panel:

  1. Select the index to apply a policy
  2. Click the “Apply Policy” button
  3. For the rollover alias, enter the alias associated with the index created in Section 2

When a policy is applied to an index, it can take up to an hour for OpenDistro to apply the policy. It can also take up to an hour for a state transition in the index policy state to be processed.

Adding heartbeat alerts

When LogStash support is added to a new CADRE system, a heartbeat alert should be added to the ElasticSearch system. The alerts are established under the Alerting panel in Kibana. A monitor must first be created to count the messages being written to the syslog index for the associated CADRE system. A trigger must then be created to alert the user when some condition based on the message count has been met. To create the monitor and trigger for heartbeat monitoring, perform the following steps:

  1. Go to the Monitors panel under Alterting in Kibana
  2. Click on the Create Monitor button
  3. Fill in the name of the monitor. It should be similar to the form <system-name> Heartbeats
  4. Select “Define using visual graph” under “How do you want to define the monitor?”
  5. Define the index alias pattern for monitoring. The pattern should be similar to the form alias-<system-name>-syslog* to monitor the syslog activity.
  6. Select the @timestamp field for the Time Field
  7. The monitor should be constructed as WHEN count() OVER all documents FOR THE LAST 21 minute(s) WHERE message is not null
  8. Since the syslog daemons on each CADRE system are configured to emit a MARK message at least every 20 minutes, set the Frequency to By interval every 20 minutes.
  9. Click the Create button
  10. Go to the panel for the newly created monitor and under Triggers, click on the Create button
  11. Fill in the Trigger Name as something similar to the form <system-name> Zero Heartbeats
  12. Leave the Severity Level as the default “1”
  13. Under Trigger Condition specify “is exactly 0”
  14. Click on the Create button

Create index pattern

An index pattern must be created to view the index data stored for a particular log file. The index pattern should match the current active index for a log file as well as the rollover indices associated with the log file so that all the data can be searched in queries and displayed in visualizations. To create an index pattern, perform the following:

  1. Go to the Management / Index Patterns panel under Kibana
  2. Click on the Create Index Pattern
  3. For the index pattern, enter a string of the form <index-name>*
  4. Click on the Next Step button
  5. Select @timestamp as the time field to use
  6. Click on the Create Index Pattern button

Performing ElasticSearch Snapshots

Snapshots are taken to create back-ups of the ElasticSearch instance. The snapshots contain the index data, including the Kibana configuration and other configuration data needed to fully restore the instance. Automatic snapshots are taken and stored for free by default in an Amazon S3 bucket. However, the automatic snapshots can only be used to restore the respective managed ElasticSearch instance. To take manual snapshots for the purposes of transferring index data to new ElasticSearch instance, such as in the case of moving data to an instance on a different VPC or zone, the ElasticSearch instance must be provisioned to take manual snapshots and store them in a specified S3 bucket. This section describes the process for restoring from the default bucket and for provisioning manual snapshots, initiating them, and restoring them. For more details on snapshot management, see https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-snapshots.html

Restoring automated snapshots

Snapshots are stored in named repositories. The repository name for automated snapshots are usually similar to the form cs-automated-enc. If the data is encrypted, then the snapshots will also be encrypted and the repository name will end in enc. To list the automated snapshots, use the list-snapshots.py script. Set the snapshot_repo_name variable to cs-automated-enc to view the list of automated snapshots. The host variable should be set to the instance of the AWS ElasticSearch instance.

If the data is encrypted, then the encryption key associated with the ElasticSearch instance is used by default. Any AWS project member with AWS ElasticSearch access can view the key on the ElasticSearch service control panel, if ever needed.

Creating and restoring manual snapshots

Manual snapshots can be used to create snapshots at will and can be transferred across different AWS ElasticSearch domains or used as an alternative to the automated snapshots for restoring a domain after a catastrophic loss of data. Before manual snapshots can be performed by an ElasticSearch domain, a snapshot repository name must first be created and that repository must be bound to an S3 bucket for storage; the storage of manual snapshots incurs S3 storage costs.

Provisioning manual snapshots

The provisioning of manual snapshots only needs to be performed once, all subsequent manual snapshots will then be saved in the specified S3 bucket. First, create an S3 bucket where the manual snapshots are to be stored. After this task is complete, IAM roles will need to be created or existing roles with sufficient permissions will need to be identified that will allow the ElasticSearch snaphot mechanism access to the S3 manual snapshot bucket. For CADRE ElasticSearch snapshots, the role CADREElasticSearchSnapshotRole provides the needed IAM permissions. This role has attached to it the CADREElasticSearchSnapshotPolicy which names the S3 buckets ElasticSearch instances under the CADRE can access and operations the instances can perform on the buckets. To complete the provisioning, you can use the python script register-repo.py to create the repository name in the ElasticSearch instance and bind the name to the destination S3 bucket. The script also specifies the needed IAM snapshot role. The script issues a REST command to https://<aws-es-instance-name>/_snapshot/<repository-name>, where is a name to be chosen for the repository to reference the manual snapshots. The json payload of the command is of the following form:

{
  "type": "s3",
  "settings": {
    "bucket": "<bucket-name>",
    "endpoint": "s3.amazonaws.com", # for us-east-1
     #"region": "us-east-2", # for all other regions
    "role_arn":   
      "arn:aws:iam::<project-id>:role/CADREElasticSearchSnapshotRole"
   }
}

The <bucket-name> and <project-id> placeholders should be replaced with the name of the S3 bucket where the manual snapshots are to be stored and the <project-id> placeholder replaced with the ID of the AWS project under which the ElasticSearch instance is operating.

Initiating manual snapshots

Once the provisioning is complete, a manual snapshot can be initiated using the create-snapshot.py script. The script issues the following command to the ElasticSearch REST interface:

POST https://<aws-es-instance-name>/_snapshot/<snapshot-repo-name>/<snapshot-name>

The <snapshot-repo-name> placeholder is typically replaced with a string indicating the date and other keyword information describing the snapshot (e.g. 06-13-20-baseline). To check the snapshot status, execute the 'list-snapshots.py' script. The status of the latest snapshot should appear at the end of the list.

Restoring snapshots

It is helpful to see the available snapshots in the manual snapshot repository. To list them, use the 'list-snapshots.py' script with the host variable set to the AWS ElasticSearch instance name and the snapshot_repo_name variable set to snapshot-repo.

If indices already exist on the instance on which the snapshot is being restored, then those with the same names as indices in the snapshot will need to be removed. The development tools in the Kibana interface should be used to issue index deletion commands due to OpenDistro security restrictions prevention delete actions from being execute remotely through the REST interface. All indices can be removed using the development tools command:

DELETE https://<aws-es-instance-name>/_all

Specific indices can be removed using:

DELETE <index-name>

Once the wanted snapshot is identified and duplicate snapshots have been removed from the instance, restore the snapshot using the command:

POST _snapshot/<snapshot-repo-name>/_restore

Setting up Kibana SSH Tunnel on Bastion

To access the Kibana ElasticSearch management console, an SSH tunnel must be established on the CADRE bastion to route traffic to Kibana web server. The address of the Kibana interface on the Ohio development VPC is:

vpc-logging-m26y7nwzhumeo5kt5ddakbrt5a.us-east-2.es.amazonaws.com

The tunnel can be established on the bastion with a command of the following form:

ssh -i ~/keys/cadre-vpc-bastion.pem -N ec2-user@<bastion-public-ip> -L <bastion-private-ip>:<bastion-port>:<kibana-address>:443

The placeholders are self explanatory. The script tunnel_to_kibana2 which resides on the bastion can be executed in the background to establish the tunnel. The security group rules should be updated to access the tunnel from the browser client IP before accessing the Kibana interface. To access the Kibana interface, point the browser to:

https://<bastion-public-ip>:<bastion-port>/_plugin/kibana/app/kibana#/home
Clone this wiki locally