Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new output module for Splunk #1091

Merged
merged 6 commits into from
Feb 20, 2024
Merged

Add new output module for Splunk #1091

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f7bdaba
Add splunk.py output module
w0Tx Feb 14, 2024
5f7596e
Add doc for splunk output module
w0Tx Feb 14, 2024
a4f030d
splunk tests
TheTechromancer Feb 17, 2024
4b94b75
updated docs
TheTechromancer Feb 17, 2024
2e70522
add output module list in README
TheTechromancer Feb 18, 2024
6967bce
update output module docs
TheTechromancer Feb 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,4 +280,26 @@ For a full list of modules, including the data types consumed and emitted by eac
| subdomain-hijack | 1 | Detects hijackable subdomains | subdomain_hijack |
| web-screenshots | 1 | Takes screenshots of web pages | gowitness |
<!-- END BBOT MODULE FLAGS -->
</details>

## BBOT Output Modules
BBOT can save its data to TXT, CSV, JSON, and tons of other destinations including [Neo4j](https://www.blacklanternsecurity.com/bbot/scanning/output/#neo4j), [Splunk](https://www.blacklanternsecurity.com/bbot/scanning/output/#splunk), and [Discord](https://www.blacklanternsecurity.com/bbot/scanning/output/#discord-slack-teams). For instructions on how to use these, see [Output Modules](https://www.blacklanternsecurity.com/bbot/scanning/output).

<!-- BBOT OUTPUT MODULES -->
| Module | Type | Needs API Key | Description | Flags | Consumed Events | Produced Events |
|-----------------|--------|-----------------|-----------------------------------------------------------------------------------------|----------------|--------------------------------------------------------------------------------------------------|---------------------------|
| asset_inventory | output | No | Merge hosts, open ports, technologies, findings, etc. into a single asset inventory CSV | | DNS_NAME, FINDING, HTTP_RESPONSE, IP_ADDRESS, OPEN_TCP_PORT, TECHNOLOGY, URL, VULNERABILITY, WAF | IP_ADDRESS, OPEN_TCP_PORT |
| csv | output | No | Output to CSV | | * | |
| discord | output | No | Message a Discord channel when certain events are encountered | | * | |
| emails | output | No | Output any email addresses found belonging to the target domain | email-enum | EMAIL_ADDRESS | |
| http | output | No | Send every event to a custom URL via a web request | | * | |
| human | output | No | Output to text | | * | |
| json | output | No | Output to Newline-Delimited JSON (NDJSON) | | * | |
| neo4j | output | No | Output to Neo4j | | * | |
| python | output | No | Output via Python API | | * | |
| slack | output | No | Message a Slack channel when certain events are encountered | | * | |
| splunk | output | No | Send every event to a splunk instance through HTTP Event Collector | | * | |
| subdomains | output | No | Output only resolved, in-scope subdomains | subdomain-enum | DNS_NAME, DNS_NAME_UNRESOLVED | |
| teams | output | No | Message a Teams channel when certain events are encountered | | * | |
| web_report | output | No | Create a markdown report with web assets | | FINDING, TECHNOLOGY, URL, VHOST, VULNERABILITY | |
| websocket | output | No | Output to websockets | | * | |
<!-- END BBOT OUTPUT MODULES -->
59 changes: 59 additions & 0 deletions bbot/modules/output/splunk.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
from bbot.core.errors import RequestError

from bbot.modules.output.base import BaseOutputModule


class Splunk(BaseOutputModule):
watched_events = ["*"]
meta = {"description": "Send every event to a splunk instance through HTTP Event Collector"}
options = {
"url": "",
"hectoken": "",
"index": "",
"source": "",
"timeout": 10,
}
options_desc = {
"url": "Web URL",
"hectoken": "HEC Token",
"index": "Index to send data to",
"source": "Source path to be added to the metadata",
"timeout": "HTTP timeout",
}

async def setup(self):
self.url = self.config.get("url", "")
self.source = self.config.get("source", "bbot")
self.index = self.config.get("index", "main")
self.timeout = self.config.get("timeout", 10)
self.headers = {}

hectoken = self.config.get("hectoken", "")
if hectoken:
self.headers["Authorization"] = f"Splunk {hectoken}"
if not self.url:
return False, "Must set URL"
if not self.source:
self.warning("Please provide a source")
return True

async def handle_event(self, event):
while 1:
try:
data = {
"index": self.index,
"source": self.source,
"sourcetype": "_json",
"event": event.json(),
}
await self.helpers.request(
url=self.url,
method="POST",
headers=self.headers,
json=data,
raise_error=True,
)
break
except RequestError as e:
self.warning(f"Error sending {event}: {e}, retrying...")
await self.helpers.sleep(1)
5 changes: 5 additions & 0 deletions bbot/scripts/docs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,11 @@ def update_individual_module_options():
assert len(bbot_module_table.splitlines()) > 50
update_md_files("BBOT MODULES", bbot_module_table)

# BBOT output modules
bbot_output_module_table = module_loader.modules_table(mod_type="output")
assert len(bbot_output_module_table.splitlines()) > 10
update_md_files("BBOT OUTPUT MODULES", bbot_output_module_table)

# BBOT module options
bbot_module_options_table = module_loader.modules_options_table()
assert len(bbot_module_options_table.splitlines()) > 100
Expand Down
58 changes: 58 additions & 0 deletions bbot/test/test_step_2/module_tests/test_module_splunk.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
import json
import httpx

from .base import ModuleTestBase


class TestSplunk(ModuleTestBase):
downstream_url = "https://splunk.blacklanternsecurity.fakedomain:1234/services/collector"
config_overrides = {
"output_modules": {
"splunk": {
"url": downstream_url,
"hectoken": "HECTOKEN",
"index": "bbot_index",
"source": "bbot_source",
}
}
}

def verify_data(self, j):
if not j["source"] == "bbot_source":
return False
if not j["index"] == "bbot_index":
return False
data = j["event"]
if not data["data"] == "blacklanternsecurity.com" and data["type"] == "DNS_NAME":
return False
return True

async def setup_after_prep(self, module_test):
self.url_correct = False
self.method_correct = False
self.got_event = False
self.headers_correct = False

async def custom_callback(request):
j = json.loads(request.content)
if request.url == self.downstream_url:
self.url_correct = True
if request.method == "POST":
self.method_correct = True
if "Authorization" in request.headers:
self.headers_correct = True
if self.verify_data(j):
self.got_event = True
return httpx.Response(
status_code=200,
)

module_test.httpx_mock.add_callback(custom_callback)
module_test.httpx_mock.add_callback(custom_callback)
module_test.httpx_mock.add_response()

def check(self, module_test, events):
assert self.got_event == True
assert self.headers_correct == True
assert self.method_correct == True
assert self.url_correct == True
1 change: 1 addition & 0 deletions docs/modules/list_of_modules.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@
| neo4j | output | No | Output to Neo4j | | * | |
| python | output | No | Output via Python API | | * | |
| slack | output | No | Message a Slack channel when certain events are encountered | | * | |
| splunk | output | No | Send every event to a splunk instance through HTTP Event Collector | | * | |
| subdomains | output | No | Output only resolved, in-scope subdomains | subdomain-enum | DNS_NAME, DNS_NAME_UNRESOLVED | |
| teams | output | No | Message a Teams channel when certain events are encountered | | * | |
| web_report | output | No | Create a markdown report with web assets | | FINDING, TECHNOLOGY, URL, VHOST, VULNERABILITY | |
Expand Down
16 changes: 5 additions & 11 deletions docs/scanning/advanced.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,10 @@ asyncio.run(main())

<!-- BBOT HELP OUTPUT -->
```text
usage: bbot [-h] [--help-all] [-t TARGET [TARGET ...]]
[-w WHITELIST [WHITELIST ...]] [-b BLACKLIST [BLACKLIST ...]]
[--strict-scope] [-m MODULE [MODULE ...]] [-l]
[-em MODULE [MODULE ...]] [-f FLAG [FLAG ...]] [-lf]
[-rf FLAG [FLAG ...]] [-ef FLAG [FLAG ...]]
[-om MODULE [MODULE ...]] [--allow-deadly] [-n SCAN_NAME]
[-o DIR] [-c [CONFIG ...]] [-v] [-d] [-s] [--force] [-y]
[--dry-run] [--current-config]
[--no-deps | --force-deps | --retry-deps | --ignore-failed-deps | --install-all-deps]
[-a] [--version]
usage: bbot [-h] [--help-all] [-t TARGET [TARGET ...]] [-w WHITELIST [WHITELIST ...]] [-b BLACKLIST [BLACKLIST ...]] [--strict-scope] [-m MODULE [MODULE ...]] [-l]
[-em MODULE [MODULE ...]] [-f FLAG [FLAG ...]] [-lf] [-rf FLAG [FLAG ...]] [-ef FLAG [FLAG ...]] [-om MODULE [MODULE ...]] [--allow-deadly] [-n SCAN_NAME] [-o DIR]
[-c [CONFIG ...]] [-v] [-d] [-s] [--force] [-y] [--dry-run] [--current-config] [--no-deps | --force-deps | --retry-deps | --ignore-failed-deps | --install-all-deps] [-a]
[--version]

Bighuge BLS OSINT Tool

Expand Down Expand Up @@ -73,7 +67,7 @@ Modules:
-ef FLAG [FLAG ...], --exclude-flags FLAG [FLAG ...]
Disable modules with these flags. (e.g. -ef aggressive)
-om MODULE [MODULE ...], --output-modules MODULE [MODULE ...]
Output module(s). Choices: asset_inventory,csv,discord,emails,http,human,json,neo4j,python,slack,subdomains,teams,web_report,websocket
Output module(s). Choices: asset_inventory,csv,discord,emails,http,human,json,neo4j,python,slack,splunk,subdomains,teams,web_report,websocket
--allow-deadly Enable the use of highly aggressive modules

Scan:
Expand Down
Loading
Loading