This is fork and a significant update from the initial publication on the ISC Storm Center website by Scott Jensen as a BACS paper and the scripts published in Github.
https://github.com/fkadriver/Dshield-ELK
https://isc.sans.edu/diary/DShield+Sensor+Monitoring+with+a+Docker+ELK+Stack+Guest+Diary/30118
This is a good reference on howto use DShield-SIEM for analysis: https://isc.sans.edu/diary/30962/
This provides an overview how the log collection with the DShield sensor is done.
https://github.com/bruneaug/DShield-SIEM/blob/main/Troubleshooting/DShield-SIEM-Flow.png
This docker is custom built to be used with the DShield Honeypot to collect and parse the logs and collect its data in a visual and easy to search for research purposes. The suggested installation is to install the DShield sensor in a Rasperry using PI Raspbian OS or a system running Ubuntu 20.04 LTS either in your network or in the cloud of your choice.
Note: This must be installed on a separate server as per the instruction below.
- This was tested on Ubuntu 20.04, 22.04 & 24.04 LTS
- Step 1 build Ubuntu
- Step 2 install docker and ELK
- Step 3 install and configure Filebeat on DShield Sensor(s)
Like in an enterprise, a system collecting security logs and monitoring a network like the DShield SIEM, it need to be installed on a separately server and not with the DShield sensor.
- Ubuntu 22.04 LTS Live Server 64-Bit
- Minimum 8+ GB - Recommended 12 GB RAM
- If the amount of RAM assigned to each containers (see below) is more than 2GB, consider increasing the server RAM capacity.
- If the amount of RAM assigned to each containers (see below) is more than 2GB, consider increasing the server RAM capacity.
- 4-8 Cores
- Add 2 partitions, one for the OS, the other for docker
- Minimum 300 GB partition assigned to /var/lib/docker
- After Ubuntu is rebooted, setup the docker partition
- Adding a 300 GB to a VM: https://github.com/bruneaug/DShield-SIEM/blob/main/AddOn/Build_a_Docker_Partition.md
- Kibana
- Elasticsearch
- Logstash
- Elastic-Agent
Note: To update ELK server components, follow these steps:
https://github.com/bruneaug/DShield-SIEM/blob/main/README.md#download-github-update
Install docker user user ($) account:
sudo apt-get install ca-certificates curl gnupg network-manager txt2html
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
Note: You may need to take the next line, remove the backlash () and put everythin in a single line in Notepad to run this echo.
echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt upgrade
sudo reboot (if update were applied)
sudo apt-get install -y jq docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin pip
sudo systemctl enable docker
Add the following to /etc/hosts where the IP address is the IP of your own ELK server:
192.168.25.231 fleet-server
192.168.25.231 es01
Using user ($) account, run the following commands:
git clone https://github.com/bruneaug/DShield-SIEM.git
chmod 754 ~/DShield-SIEM/scripts/cowrie-setup.sh
mkdir scripts
mv DShield-SIEM/AddOnScripts/parsing_tty.sh scripts
mv DShield-SIEM/AddOnScripts/rename_arkime_pcap.sh scripts
chmod 754 scripts/*.sh
The parsing_tty.sh script will be configured later in another document.
cd ~/DShield-SIEM
Note: Before installation, you can edit the .env (ls -la to see it) file to make any derided changes.
- Current default password for elastic is student
Memory Limits in .env are the most memory that docker will allocate for each of the ELK containers.
Default to 2147483648 (2GB) but can be expanded if you have the resources
- Update the following variables in the .env file to match your ELK server DNS information, hostname, IP and default elastic password if you want to change it:
- HOSTNAME="ubuntu"
- DNS_SERVER="9.9.9.9"
- IPADDRESS="192.168.25.231"
- ELASTIC_PASSWORD=student
- If you want to change the default nameserver(s) information and your local private network location for the destination address (i.e. DShield sensor mapping - currently set for Ottawa, Canada), edit to the following directory for the files before loading docker:
cd logstash/pipeline
- logstash-200-filter-cowrie.conf
- logstash-201-filter-iptables.conf
- logstash-202-filter-cowrie-webhoneypot.conf
You can keep these default or edit each files and change them.
Now execute docker compose to build the ELK server applications.
This will build: Kibana, Elasticsearch, elastic-agent, Logstash and load the Cowrie parsers, configuration files and dashboard.
$ sudo docker compose up -d
Enable and start the docker service. This will restart DShield-SIEM when the server is rebooted.
sudo systemctl enable docker.service
sudo systemctl start docker.service
Confirm the docker service is running
sudo systemctl status docker.service
Installation Completed
This command shows the list of docker directories in used after all of Elasticsearch components have been installed.
As data gets added to Elasticsearch, you can also monitor either with the command below or within ELK the amount of disk is available for storage.
sudo du --human-readable --max-depth 1 --no-dereference --one-file-system /var/lib/docker
Using netstat, these 4 services should now be listening.
netstat -an | grep '9200\|8220\|5601\|5044'
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN ---> Kibana
tcp 0 0 0.0.0.0:8220 0.0.0.0:* LISTEN ---> elastic-agent
tcp 0 0 0.0.0.0:9200 0.0.0.0:* LISTEN ---> Elasticsearch
tcp 0 0 0.0.0.0:5044 0.0.0.0:* LISTEN ---> Logstash
tcp6 0 0 :::5601 :::* LISTEN
tcp6 0 0 :::8220 :::* LISTEN
tcp6 0 0 :::9200 :::* LISTEN
tcp6 0 0 :::5044 :::* LISTEN
After docker finish installing all the ELK docker components, now it is time to login the ELK stack using your Ubuntu server IP.
- Login Kibana with username: elastic and default password if it hasn't been changed: student
Web Access: https://serverIP:5601
In order to see the ELK stack health, it is necessary to configure the stack monitoring by navigating to: Management -> Stack Monitoring
- Select "Or, set up with self monitoring"
- Monitoring is currently off -> Select: Turn on monitoring
The fleet-agent (this is an elastic-agent running on the ELK stack) will be used to ingest threat intelligence. It can also be used to do other things that won’t be covered here.
This is an example for the format to setup the fleet-server and the elastic-agent:
https://github.com/bruneaug/DShield-SIEM/blob/main/Troubleshooting/fleet-server-examples.txt
- From the dropdown menu, select Management → Fleet →Settings → Edit Outputs (Actions)
- Login server via SSH
- Copy ca.crt certificate to /tmp
sudo cp /var/lib/docker/volumes/dshield-elk_certs/_data/ca/ca.crt /tmp
- Get a copy of Elasticsearch CA trusted fingerprint
sudo openssl x509 -fingerprint -sha256 -noout -in /tmp/ca.crt | awk -F"=" {' print $2 '} | sed s/://g
-
The output will look like this:
673FB617E15CCCE73F9B647EF99449642A19CFC1D75BF5772047DA99DB950844 -
Get Content of Elasticsearch CA Certificate to Apply to Advanced YAML configuration. Type the command because it doesn't copy well
sudo cat /tmp/ca.crt | sed -r 's/(.*)/ \1/g'
Follow the example from the Troubleshooting fleet-server-examples guide URL above for the correct format.
sed will add the 4 spaces with the previous command against the CA certificate
- Change the Hosts to: https://es01:9200
After adding the certificate information, save and apply these settings.
Followed by Save and deploy
Under Settings, configure the Fleet server hosts by either edit or Add Fleet Server configured as follows:
Next phase is to Select Agent Policy → Add Agent → Enroll in Fleet → Add Fleet Server
- Provide a Name: es01
- Provide URL: https://fleet-server:8220
- Last: Generate Fleet Server policy
- Select: RPM
Refer to this page for an example of Fleet-Server SSL Configuration:
https://github.com/bruneaug/DShield-SIEM/blob/main/Troubleshooting/fleet-server-examples.txt
We are going to need this information to setup our fleet server.
Login via SSH to the fleet-server and make sure the fleet-server is running before setting up our agent:
sudo docker start fleet-server
sudo docker exec -ti fleet-server bash
./elastic-agent status (check it is running)
./elastic-agent restart (if it doesn't appear to be running, force a restart, and recheck the status)
This is an example of what need to be copied to the fleet server. Ensure the fleet server es is: https://es01:9200
Add the bold section after port=8220 because are certificates are self-generated. This will ensure the agent takes the update.
The token and fingerprint will be different than this example
Copy the elastic-enrol agent below and eplace these 2 lines taken from your own serve and everything else remain the same.
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MzEwOTcwODU3MzI6VjMyLU13cmFUOUM1eUFvMUhVUFl5QQ \ --fleet-server-es-ca-trusted-fingerprint=0D9A25F4C147EB3A496253525DF6F039CF3C19776E64A1F77CEFCCD08B76BC61 \
First step, don't forget to login in the fleet-agent before enrolling your fleet-agent:
sudo docker exec -ti fleet-server bash
Use this template to install your certificate
elastic-agent enroll \
--url=https://fleet-server:8220 \
--fleet-server-es=https://es01:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MzEwOTcwODU3MzI6VjMyLU13cmFUOUM1eUFvMUhVUFl5QQ \
--fleet-server-policy=fleet-server-policy \
--fleet-server-es-ca=/certs/es01/es01.crt \
--fleet-server-es-ca-trusted-fingerprint=0D9A25F4C147EB3A496253525DF6F039CF3C19776E64A1F77CEFCCD08B76BC61 \
--fleet-server-port=8220 \
--certificate-authorities=/certs/ca/ca.crt \
--fleet-server-cert=/certs/fleet-server/fleet-server.crt \
--fleet-server-cert-key=/certs/fleet-server/fleet-server.key \
--elastic-agent-cert=/certs/fleet-server/fleet-server.crt \
--elastic-agent-cert-key=/certs/fleet-server/fleet-server.key \
--fleet-server-es-cert=/certs/fleet-server/fleet-server.crt \
--fleet-server-es-cert-key=/certs/fleet-server/fleet-server.key
This will replace your current settings. Do you want to continue? [Y/n]: Y
Successfully enrolled the Elastic Agent.
From your current location, verify it installed correctly
./elastic-agent status
./elastic-agent restart (if you got what appears to be errors, force a restart and check the status)
Now that the Fleet Server is connected, close this Windows and lets proceed to the next step.
In Elastic Management → Fleet, refresh Agents and this is what shows up:
The server is now ready to install Threat Intel Agents to be used in Security (SIEM portion) against the honeypot logs.
The next step is to select Agent policies → Fleet Server Policy → Add integration:
Use this Dashboard to view the metrics collected by elastic-agent:
- Select and Add AlienVault OTX (need an API key)
- Select AbuseCH (no API key needed)
- Select Threat Intelligence Utilities
- Elasticsearch
- Select Kibana
- Select Docker
- In Elastic Management → Installed Integration
- Select each of the installed integration, then select Settings and enable the tab to keep the policy up to date:
- Select Rules → Detection rules (SIEM) → Add Elastic rules
- Under Search Tags: Rule Type: Threat Intel (add at the minimum those 4 rules)
- Install and enable those 4 rules
- Threat Intel Indicator Match - Cowrie (Custom rule loaded and enabled when docker load all the ELK stack components)
- Threat Intel Windows Registry Indicator Match
- Threat Intel Hash Indicator Match
- Threat Intel URL Indicator Match
- You can look through the rules and enable those other rules that you want to try against your honeypot data.
If some of the threat intel above were added to the elastic-agent, and cowrie* was added to the Management -> Advanced Settings during the initial installation, it can now track rules that match in the alert trends of the SIEM part of the ELK stack.
This is the output from Management → Stack Management → Advanced Settings
If any threat intel match, the SIEM will display the following activity that can now be investigated:
Next step is to add the Filebeat package to the DShield Sensor to send the logs the Elasticsearch.
Use the following steps to install Filebeat using the following commands taken from this reference [3]:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https<br>
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
echo "deb https://artifacts.elastic.co/packages/oss-8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install filebeat elastic-agent softflowd
Download the custom filebeat.yml file that will forward the logs the Elasticsearch:
sudo curl https://raw.githubusercontent.com/bruneaug/DShield-Sensor/refs/heads/main/filebeat.yml -o /etc/filebeat/filebeat.yml
- Edit the filebeat.yml and change the IP address to the logstash parser (192.168.25.23) to match the IP used by Logstash:
sudo vi /etc/filebeat/filebeat.yml
output.logstash:
hosts: ["192.168.25.23:5044"]
sudo su -
filebeat test config
Expected output: Config OK
filebeat test output
Expected output:
logstash: 192.168.25.231:5044...
connection...
parse host... OK
dns lookup... OK
addresses: 192.168.25.231
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
sudo systemctl enable filebeat
sudo systemctl start filebeat
sudo systemctl status filebeat
sudo systemctl enable elastic-agent
sudo systemctl start elastic-agent
sudo systemctl enable softflowd
sudo systemctl start softflowd
Filebeat tracks the events it has processed with a file located called log.json, if deleted, all the events that were previous sent to Elasticsearch will be reprocessed when filebeat is restarted.
The location of this file:
sudo su -
cd /var/lib/filebeat/registry/filebeat
If you are planning to resend all the logs because your ELK server got rebuild, stop filebeat, delete log.json and restart filebeat.
This will usually take a while to catch up in the ELK Stack dashboard. If you are looking to see if it is sending the logs to the ELK Stack, you could
- tail -f log.json
- Look for the past 30 days in the dashboard while it is catching up
Follow this step-by-step documents to install the elastic-agent to the DShield sensor
https://github.com/bruneaug/DShield-SIEM/blob/main/Configure-Elastic-Agent.pdf
To access the Dashboard select Analytics -> Dashboard -> [Logs DShield Sensor] Overview
Manual restart of the docker
cd DShield-SIEM
sudo docker compose stop
sudo docker compose start
If you sudo systemctl enable docker during the setup above
Docker will automatically start all the docker services. If you need to restart the docker service, use either of these commands:
sudo systemctl restart docker or
sudo reboot the server
I have move the list of commands to its own page
Refer to this page: https://github.com/bruneaug/DShield-SIEM/edit/main/Troubleshooting/docker_useful_commands..md
These command are run from the Dev Tool -> Console
GET _nodes/http?pretty (Get a list and information of all the nodes)
GET _security/_authenticate
[1] https://isc.sans.edu/tools/honeypot/
[2] https://www.elastic.co/downloads/beats/filebeat
[3] https://www.elastic.co/guide/en/beats/filebeat/8.8/setup-repositories.html#_apt
[4] https://isc.sans.edu/diary/DShield+Honeypot+Activity+for+May+2023/29932
[5] https://isc.sans.edu/diary/DShield+Sensor+JSON+Log+to+Elasticsearch/29458
[6] https://isc.sans.edu/diary/DShield+Sensor+JSON+Log+Analysis/29412
[7] https://github.com/jslagrew/cowrieprocessor/blob/main/submit_vtfiles.py
[8] https://handlers.sans.edu/gbruneau/elastic.htm
[9] https://www.elastic.co/guide/en/fleet/current/secure-connections.html
[10] https://www.docker.elastic.co/