Skip to content

Commit

Permalink
merge dev
Browse files Browse the repository at this point in the history
  • Loading branch information
@TheTechromancer
TheTechromancer committed Feb 20, 2024
2 parents ac5ac64 + 069ecff commit e6aef5a
Show file tree
Hide file tree
Showing 15 changed files with 264 additions and 30 deletions.
36 changes: 32 additions & 4 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

### A Recursive Internet Scanner for Hackers.

[![Python Version](https://img.shields.io/badge/python-3.9+-FF8400)](https://www.python.org) [![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![License](https://img.shields.io/badge/license-GPLv3-FF8400.svg)](https://github.com/blacklanternsecurity/bbot/blob/dev/LICENSE) [![DEF CON Demo Labs 2023](https://img.shields.io/badge/DEF%20CON%20Demo%20Labs-2023-FF8400.svg)](https://forum.defcon.org/node/246338) [![Tests](https://github.com/blacklanternsecurity/bbot/actions/workflows/tests.yml/badge.svg?branch=stable)](https://github.com/blacklanternsecurity/bbot/actions?query=workflow%3A"tests") [![Codecov](https://codecov.io/gh/blacklanternsecurity/bbot/branch/dev/graph/badge.svg?token=IR5AZBDM5K)](https://codecov.io/gh/blacklanternsecurity/bbot) [![Pypi Downloads](https://img.shields.io/pypi/dm/bbot)](https://pypistats.org/packages/bbot) [![Discord](https://img.shields.io/discord/859164869970362439)](https://discord.com/invite/PZqkgxu5SA)
[![Python Version](https://img.shields.io/badge/python-3.9+-FF8400)](https://www.python.org) [![License](https://img.shields.io/badge/license-GPLv3-FF8400.svg)](https://github.com/blacklanternsecurity/bbot/blob/dev/LICENSE) [![DEF CON Demo Labs 2023](https://img.shields.io/badge/DEF%20CON%20Demo%20Labs-2023-FF8400.svg)](https://forum.defcon.org/node/246338) [![PyPi Downloads](https://static.pepy.tech/personalized-badge/bbot?right_color=orange&left_color=grey)](https://pepy.tech/project/bbot) [![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![Tests](https://github.com/blacklanternsecurity/bbot/actions/workflows/tests.yml/badge.svg?branch=stable)](https://github.com/blacklanternsecurity/bbot/actions?query=workflow%3A"tests") [![Codecov](https://codecov.io/gh/blacklanternsecurity/bbot/branch/dev/graph/badge.svg?token=IR5AZBDM5K)](https://codecov.io/gh/blacklanternsecurity/bbot) [![Discord](https://img.shields.io/discord/859164869970362439)](https://discord.com/invite/PZqkgxu5SA)

BBOT (Bighuge BLS OSINT Tool) is a recursive internet scanner inspired by [Spiderfoot](https://github.com/smicallef/spiderfoot), but designed to be faster, more reliable, and friendlier to pentesters, bug bounty hunters, and developers.

Expand Down Expand Up @@ -62,7 +62,7 @@ git clone https://github.com/blacklanternsecurity/bbot && cd bbot
</details>

<details>
<summary><b>Usage</b></summary>
<summary><b>Example Usage</b></summary>

## Example Commands

Expand Down Expand Up @@ -114,7 +114,13 @@ bbot -t evilcorp.com -f subdomain-enum email-enum cloud-enum web-basic -m nmap g

## Targets

BBOT accepts an unlimited number of targets via `-t`. You can specify targets either directly on the command line or in files (or both!). Targets can be any of the following:
BBOT accepts an unlimited number of targets via `-t`. You can specify targets either directly on the command line or in files (or both!):

```bash
bbot -t evilcorp.com evilcorp.org 1.2.3.0/24 -f subdomain-enum
```

Targets can be any of the following:

- `DNS_NAME` (`evilcorp.com`)
- `IP_ADDRESS` (`1.2.3.4`)
Expand Down Expand Up @@ -280,4 +286,26 @@ For a full list of modules, including the data types consumed and emitted by eac
| subdomain-hijack | 1 | Detects hijackable subdomains | subdomain_hijack |
| web-screenshots | 1 | Takes screenshots of web pages | gowitness |
<!-- END BBOT MODULE FLAGS -->
</details>

## BBOT Output Modules
BBOT can save its data to TXT, CSV, JSON, and tons of other destinations including [Neo4j](https://www.blacklanternsecurity.com/bbot/scanning/output/#neo4j), [Splunk](https://www.blacklanternsecurity.com/bbot/scanning/output/#splunk), and [Discord](https://www.blacklanternsecurity.com/bbot/scanning/output/#discord-slack-teams). For instructions on how to use these, see [Output Modules](https://www.blacklanternsecurity.com/bbot/scanning/output).

<!-- BBOT OUTPUT MODULES -->
| Module | Type | Needs API Key | Description | Flags | Consumed Events | Produced Events |
|-----------------|--------|-----------------|-----------------------------------------------------------------------------------------|----------------|--------------------------------------------------------------------------------------------------|---------------------------|
| asset_inventory | output | No | Merge hosts, open ports, technologies, findings, etc. into a single asset inventory CSV | | DNS_NAME, FINDING, HTTP_RESPONSE, IP_ADDRESS, OPEN_TCP_PORT, TECHNOLOGY, URL, VULNERABILITY, WAF | IP_ADDRESS, OPEN_TCP_PORT |
| csv | output | No | Output to CSV | | * | |
| discord | output | No | Message a Discord channel when certain events are encountered | | * | |
| emails | output | No | Output any email addresses found belonging to the target domain | email-enum | EMAIL_ADDRESS | |
| http | output | No | Send every event to a custom URL via a web request | | * | |
| human | output | No | Output to text | | * | |
| json | output | No | Output to Newline-Delimited JSON (NDJSON) | | * | |
| neo4j | output | No | Output to Neo4j | | * | |
| python | output | No | Output via Python API | | * | |
| slack | output | No | Message a Slack channel when certain events are encountered | | * | |
| splunk | output | No | Send every event to a splunk instance through HTTP Event Collector | | * | |
| subdomains | output | No | Output only resolved, in-scope subdomains | subdomain-enum | DNS_NAME, DNS_NAME_UNRESOLVED | |
| teams | output | No | Message a Teams channel when certain events are encountered | | * | |
| web_report | output | No | Create a markdown report with web assets | | FINDING, TECHNOLOGY, URL, VHOST, VULNERABILITY | |
| websocket | output | No | Output to websockets | | * | |
<!-- END BBOT OUTPUT MODULES -->
22 changes: 16 additions & 6 deletions bbot/core/event/base.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -565,7 +565,7 @@ def __contains__(self, other):
return host_in_host(other.host, self.host)
return False

def json(self, mode="json"):
def json(self, mode="json", siem_friendly=False):
"""
Serializes the event object to a JSON-compatible dictionary.
Expand All @@ -574,6 +574,7 @@ def json(self, mode="json"):
Parameters:
mode (str): Specifies the data serialization mode. Default is "json". Other options include "graph", "human", and "id".
siem_friendly (bool): Whether to format the JSON in a way that's friendly to SIEM ingestion by Elastic, Splunk, etc. This ensures the value of "data" is always the same type (a dictionary).
Returns:
dict: JSON-serializable dictionary representation of the event object.
Expand All @@ -585,9 +586,13 @@ def json(self, mode="json"):
j.update({i: v})
data_attr = getattr(self, f"data_{mode}", None)
if data_attr is not None:
j["data"] = data_attr
data = data_attr
else:
j["data"] = smart_decode(self.data)
data = smart_decode(self.data)
if siem_friendly:
j["data"] = {self.type: data}
else:
j["data"] = data
web_spider_distance = getattr(self, "web_spider_distance", None)
if web_spider_distance is not None:
j["web_spider_distance"] = web_spider_distance
Expand Down Expand Up @@ -1312,7 +1317,7 @@ def make_event(
)


def event_from_json(j):
def event_from_json(j, siem_friendly=False):
"""
Creates an event object from a JSON dictionary.
Expand All @@ -1335,14 +1340,19 @@ def event_from_json(j):
if required keys are missing. Make sure to validate the JSON input beforehand.
"""
try:
event_type = j["type"]
kwargs = {
"data": j["data"],
"event_type": j["type"],
"event_type": event_type,
"scans": j.get("scans", []),
"tags": j.get("tags", []),
"confidence": j.get("confidence", 5),
"dummy": True,
}
if siem_friendly:
data = j["data"][event_type]
else:
data = j["data"]
kwargs["data"] = data
event = make_event(**kwargs)

resolved_hosts = j.get("resolved_hosts", [])
Expand Down
5 changes: 3 additions & 2 deletions bbot/core/helpers/interactsh.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,9 @@ async def register(self, callback=None):
if self.custom_server:
if not self.token:
log.verbose("Interact.sh token is not set")
headers["Authorization"] = self.token
self.server_list = [self.custom_server]
else:
headers["Authorization"] = self.token
self.server_list = [str(self.custom_server)]
else:
self.server_list = random.sample(server_list, k=len(server_list))
for server in self.server_list:
Expand Down
5 changes: 4 additions & 1 deletion bbot/modules/output/http.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ class HTTP(BaseOutputModule):
"username": "",
"password": "",
"timeout": 10,
"siem_friendly": False,
}
options_desc = {
"url": "Web URL",
Expand All @@ -21,12 +22,14 @@ class HTTP(BaseOutputModule):
"username": "Username (basic auth)",
"password": "Password (basic auth)",
"timeout": "HTTP timeout",
"siem_friendly": "Format JSON in a SIEM-friendly way for ingestion into Elastic, Splunk, etc.",
}

async def setup(self):
self.url = self.config.get("url", "")
self.method = self.config.get("method", "POST")
self.timeout = self.config.get("timeout", 10)
self.siem_friendly = self.config.get("siem_friendly", False)
self.headers = {}
bearer = self.config.get("bearer", "")
if bearer:
Expand All @@ -52,7 +55,7 @@ async def handle_event(self, event):
method=self.method,
auth=self.auth,
headers=self.headers,
json=dict(event),
json=event.json(siem_friendly=self.siem_friendly),
raise_error=True,
)
break
Expand Down
4 changes: 1 addition & 3 deletions bbot/modules/output/json.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,7 @@ async def setup(self):
return True

async def handle_event(self, event):
event_json = dict(event)
if self.siem_friendly:
event_json["data"] = {event.type: event_json.pop("data", "")}
event_json = event.json(siem_friendly=self.siem_friendly)
event_str = json.dumps(event_json)
if self.file is not None:
self.file.write(event_str + "\n")
Expand Down
59 changes: 59 additions & 0 deletions bbot/modules/output/splunk.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
from bbot.core.errors import RequestError

from bbot.modules.output.base import BaseOutputModule


class Splunk(BaseOutputModule):
watched_events = ["*"]
meta = {"description": "Send every event to a splunk instance through HTTP Event Collector"}
options = {
"url": "",
"hectoken": "",
"index": "",
"source": "",
"timeout": 10,
}
options_desc = {
"url": "Web URL",
"hectoken": "HEC Token",
"index": "Index to send data to",
"source": "Source path to be added to the metadata",
"timeout": "HTTP timeout",
}

async def setup(self):
self.url = self.config.get("url", "")
self.source = self.config.get("source", "bbot")
self.index = self.config.get("index", "main")
self.timeout = self.config.get("timeout", 10)
self.headers = {}

hectoken = self.config.get("hectoken", "")
if hectoken:
self.headers["Authorization"] = f"Splunk {hectoken}"
if not self.url:
return False, "Must set URL"
if not self.source:
self.warning("Please provide a source")
return True

async def handle_event(self, event):
while 1:
try:
data = {
"index": self.index,
"source": self.source,
"sourcetype": "_json",
"event": event.json(),
}
await self.helpers.request(
url=self.url,
method="POST",
headers=self.headers,
json=data,
raise_error=True,
)
break
except RequestError as e:
self.warning(f"Error sending {event}: {e}, retrying...")
await self.helpers.sleep(1)
5 changes: 5 additions & 0 deletions bbot/scripts/docs.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,11 @@ def update_individual_module_options():
assert len(bbot_module_table.splitlines()) > 50
update_md_files("BBOT MODULES", bbot_module_table)

# BBOT output modules
bbot_output_module_table = module_loader.modules_table(mod_type="output")
assert len(bbot_output_module_table.splitlines()) > 10
update_md_files("BBOT OUTPUT MODULES", bbot_output_module_table)

# BBOT module options
bbot_module_options_table = CORE.module_loader.modules_options_table()
assert len(bbot_module_options_table.splitlines()) > 100
Expand Down
13 changes: 13 additions & 0 deletions bbot/test/test_step_1/test_events.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,19 @@ async def test_events(events, scan, helpers, bbot_config):
assert reconstituted_event.type == "DNS_NAME"
assert "127.0.0.1" in reconstituted_event.resolved_hosts

# SIEM-friendly serialize/deserialize
json_event_siemfriendly = db_event.json(siem_friendly=True)
assert json_event_siemfriendly["scope_distance"] == 1
assert json_event_siemfriendly["data"] == {"DNS_NAME": "evilcorp.com"}
assert json_event_siemfriendly["type"] == "DNS_NAME"
assert json_event_siemfriendly["timestamp"] == timestamp
reconstituted_event2 = event_from_json(json_event_siemfriendly, siem_friendly=True)
assert reconstituted_event2.scope_distance == 1
assert reconstituted_event2.timestamp.timestamp() == timestamp
assert reconstituted_event2.data == "evilcorp.com"
assert reconstituted_event2.type == "DNS_NAME"
assert "127.0.0.1" in reconstituted_event2.resolved_hosts

http_response = scan.make_event(httpx_response, "HTTP_RESPONSE", source=scan.root_event)
assert http_response.source_id == scan.root_event.id
assert http_response.data["input"] == "http://example.com:80"
Expand Down
Loading

0 comments on commit e6aef5a

Please sign in to comment.