diff --git a/README.md b/README.md index a5e4f3c..0a301c7 100644 --- a/README.md +++ b/README.md @@ -159,7 +159,7 @@ If you want to get a (free) token for your systems - feel free to contact us at: A simple script that follows the content of a specific log-file and parses abuser information from it. -See: [Report Script](https://github.com/O-X-L/risk-db/blob/latest/report_script/README.md) +See: [Report Script](https://github.com/O-X-L/risk-db/blob/latest/reporting/README.md) #### Fail2Ban diff --git a/reporting/Graylog.md b/reporting/Graylog.md new file mode 100644 index 0000000..c0db4f0 --- /dev/null +++ b/reporting/Graylog.md @@ -0,0 +1,173 @@ +# Reporting via Graylog Alerts + +We can create a Graylog Alert Notification to Report Abusers to this Risk-Database. + +You can find an example on how to split HAProxy logs into different fields here: [gist.github.com](https://gist.github.com/superstes/a2f6c5d855857e1f10dcb51255fe08c6#haproxy-split) (*via Pipeline Rules*) + +## API Service + +As Graylog has no option to add advanced filters for the data sent by the notifications, we will have to add a minimal service to do so. + +1. Add the API Service Script: (File: `/usr/local/bin/notification-api.py`) + + ```python3 + #!/usr/bin/env python3 + + # Abuse Reporting System: https://github.com/O-X-L/risk-db + + from time import sleep + + from waitress import serve + from flask import Flask, request, jsonify, make_response + from requests import post as http_post + from requests.exceptions import HTTPError, SSLError + + DEBUG = False + PORT = 8000 + RISK_DB_TOKEN = 'XXX' + + # for internal proxy servers + # proxy_server = { + # 'http': 'http://{{ proxy_server }}:{{ proxy_server_port }}', + # 'https': 'http://{{ proxy_server }}:{{ proxy_server_port }}', + # } + proxy_server = None + + app = Flask(__name__) + + + @app.route('/report-abuse/haproxy', methods=['POST']) + def report_abuse_haproxy(): + unique_list = [] + + for log in request.json['backlog']: + try: + # todo: update the field names to the ones you use + log = log['fields'] + + # todo: add additional filters here and 'continue' if a log should be skipped + + ip = log['haproxy_client'] + + # making sure we do not report a client twice (every minute) + if ip in unique_list: + continue + + unique_list.append(ip) + + cause = 'probe' + # todo: add conditions to decide on what category to report on + if str(log['xyz']) != '1': + cause = 'bot' + + elif str(log['haproxy_status']) == '429': + cause = 'rate' + + ua = log['haproxy_user_agent'][:50] + + print(f'REPORTING: {ip} because of {cause} ("{ua}")') + http_post( + url='https://risk.oxl.app/api/report', + proxies=proxy_server, + headers={'Token': RISK_DB_TOKEN}, + json={ + 'ip': ip, + 'cat': cause, + 'cmt': ua, + }, + ) + + if response.status_code != 200 and DEBUG: + print('Abuse-Report failed', response.status_code) + + sleep(1) + + except (KeyError, HTTPError, SSLError): + continue + + return make_response(jsonify({}), 200) + + + if __name__ == '__main__': + serve(app, host='127.0.0.1', port=PORT) + ``` + +2. Add a python3 virtual environment: + + ```bash + apt install python3-virtualenv + python3 -m virtualenv /var/local/graylog-notification-api/venv + source /var/local/graylog-notification-api/venv/bin/activate + pip install requests flask waitress + ``` + +3. Add a systemd service to run the api script: (File: `/etc/systemd/system/graylog-notification-api.service`) + + ``` + [Unit] + Description=Graylog Notification API Service + + [Service] + Type=simple + User=graylog + Group=graylog + Environment=PYTHONUNBUFFERED=1 + ExecStart=/bin/bash -c "source /var/local/graylog-notification-api/venv/bin/activate && python3 /usr/local/bin/notification-api.py" + + StandardOutput=journal + StandardError=journal + SyslogIdentifier=notification-api + Restart=on-failure + RestartSec=10s + TimeoutStopSec=30s + + [Install] + WantedBy=multi-user.target + ``` + +4. Enable and start the service: + + ```bash + systemctl daemon-reload + systemctl start graylog-notification-api.service + systemctl enable graylog-notification-api.service + ``` + +---- + +## Graylog Alerts + +### Create an Alert-Notification + +`https:///alerts/notifications` + +* **Title**: `Report Abuse - HAProxy` +* **Notification Type**: `HTTP Notification` +* **URL**: `http://127.0.0.1:8000/report-abuse/haproxy` + + +### Create an Alert-Event + +`https:///alerts/definitions` + +**Event Details**: + + * **Title**: `HAProxy Abuse` + * **Priority**: `Low` + +**Condition**: + + * **Condition Type**: `Filter & Aggregation` + * **Streams**: Select your HAProxy Access-Log stream + * **Search Query**: Filter Logs to only include blocks of your security filters. Also exclude your `safe-ips` and so on + * **Search within the last**: 1 minute + * **Execute search every**: 1 minute + * **Create Events for Definition if...**: `Aggregation of results reaches a threshold` + * **Execute search every**: 1 minute + * **If**: `count()` **is** `>` **Threshold** `0` + +**Notifications**: + + * **Choose Notification**: `Report Abuse - HAProxy` + * **Grace Period**: Disable + * **Message Backlog**: 500 (duplicates will be filtered by the API-service) diff --git a/report_script/README.md b/reporting/README.md similarity index 94% rename from report_script/README.md rename to reporting/README.md index c23c629..1cbf11d 100644 --- a/report_script/README.md +++ b/reporting/README.md @@ -6,7 +6,7 @@ Note: Bash regex does not support PCRE like `\d`, `\s` nor non-greedy `*?` You can run this script as systemd service: -* Example: [report_log_watcher.service](https://github.com/O-X-L/risk-db/blob/latest/report_script/log_watcher_APP.service) +* Example: [report_log_watcher.service](https://github.com/O-X-L/risk-db/blob/latest/reporting/log_watcher_APP.service) * Copy the scripts to `/usr/local/bin/` (*or wherever you want them to be*) * Enable/Start: @@ -45,7 +45,7 @@ See the scripts for examples. ### NFTables Firewall -See: [Report Script - NFTables](https://github.com/O-X-L/risk-db/blob/latest/report_script/log_watcher_nftables.sh) +See: [Report Script - NFTables](https://github.com/O-X-L/risk-db/blob/latest/reporting/log_watcher_nftables.sh) #### Logs @@ -94,7 +94,7 @@ Content: ### HAProxy -See: [Report Script - HAProxy](https://github.com/O-X-L/risk-db/blob/latest/report_script/log_watcher_haproxy.sh) +See: [Report Script - HAProxy](https://github.com/O-X-L/risk-db/blob/latest/reporting/log_watcher_haproxy.sh) #### Logs diff --git a/report_script/in_ip_list.py b/reporting/in_ip_list.py similarity index 100% rename from report_script/in_ip_list.py rename to reporting/in_ip_list.py diff --git a/report_script/log_watcher_APP.service b/reporting/log_watcher_APP.service similarity index 100% rename from report_script/log_watcher_APP.service rename to reporting/log_watcher_APP.service diff --git a/report_script/log_watcher_haproxy.sh b/reporting/log_watcher_haproxy.sh similarity index 100% rename from report_script/log_watcher_haproxy.sh rename to reporting/log_watcher_haproxy.sh diff --git a/report_script/log_watcher_nftables.sh b/reporting/log_watcher_nftables.sh similarity index 100% rename from report_script/log_watcher_nftables.sh rename to reporting/log_watcher_nftables.sh