Skip to content

Commit

Permalink
Merge branch 'dev' into baddns_module
Browse files Browse the repository at this point in the history
  • Loading branch information
@liquidsec
liquidsec authored Feb 20, 2024
2 parents 25faf8c + 069ecff commit bc57ab5
Show file tree
Hide file tree
Showing 51 changed files with 859 additions and 443 deletions.
36 changes: 32 additions & 4 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

### A Recursive Internet Scanner for Hackers.

[![Python Version](https://img.shields.io/badge/python-3.9+-FF8400)](https://www.python.org) [![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![License](https://img.shields.io/badge/license-GPLv3-FF8400.svg)](https://github.com/blacklanternsecurity/bbot/blob/dev/LICENSE) [![DEF CON Demo Labs 2023](https://img.shields.io/badge/DEF%20CON%20Demo%20Labs-2023-FF8400.svg)](https://forum.defcon.org/node/246338) [![Tests](https://github.com/blacklanternsecurity/bbot/actions/workflows/tests.yml/badge.svg?branch=stable)](https://github.com/blacklanternsecurity/bbot/actions?query=workflow%3A"tests") [![Codecov](https://codecov.io/gh/blacklanternsecurity/bbot/branch/dev/graph/badge.svg?token=IR5AZBDM5K)](https://codecov.io/gh/blacklanternsecurity/bbot) [![Pypi Downloads](https://img.shields.io/pypi/dm/bbot)](https://pypistats.org/packages/bbot) [![Discord](https://img.shields.io/discord/859164869970362439)](https://discord.com/invite/PZqkgxu5SA)
[![Python Version](https://img.shields.io/badge/python-3.9+-FF8400)](https://www.python.org) [![License](https://img.shields.io/badge/license-GPLv3-FF8400.svg)](https://github.com/blacklanternsecurity/bbot/blob/dev/LICENSE) [![DEF CON Demo Labs 2023](https://img.shields.io/badge/DEF%20CON%20Demo%20Labs-2023-FF8400.svg)](https://forum.defcon.org/node/246338) [![PyPi Downloads](https://static.pepy.tech/personalized-badge/bbot?right_color=orange&left_color=grey)](https://pepy.tech/project/bbot) [![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![Tests](https://github.com/blacklanternsecurity/bbot/actions/workflows/tests.yml/badge.svg?branch=stable)](https://github.com/blacklanternsecurity/bbot/actions?query=workflow%3A"tests") [![Codecov](https://codecov.io/gh/blacklanternsecurity/bbot/branch/dev/graph/badge.svg?token=IR5AZBDM5K)](https://codecov.io/gh/blacklanternsecurity/bbot) [![Discord](https://img.shields.io/discord/859164869970362439)](https://discord.com/invite/PZqkgxu5SA)

BBOT (Bighuge BLS OSINT Tool) is a recursive internet scanner inspired by [Spiderfoot](https://github.com/smicallef/spiderfoot), but designed to be faster, more reliable, and friendlier to pentesters, bug bounty hunters, and developers.

Expand Down Expand Up @@ -62,7 +62,7 @@ git clone https://github.com/blacklanternsecurity/bbot && cd bbot
</details>

<details>
<summary><b>Usage</b></summary>
<summary><b>Example Usage</b></summary>

## Example Commands

Expand Down Expand Up @@ -114,7 +114,13 @@ bbot -t evilcorp.com -f subdomain-enum email-enum cloud-enum web-basic -m nmap g

## Targets

BBOT accepts an unlimited number of targets via `-t`. You can specify targets either directly on the command line or in files (or both!). Targets can be any of the following:
BBOT accepts an unlimited number of targets via `-t`. You can specify targets either directly on the command line or in files (or both!):

```bash
bbot -t evilcorp.com evilcorp.org 1.2.3.0/24 -f subdomain-enum
```

Targets can be any of the following:

- `DNS_NAME` (`evilcorp.com`)
- `IP_ADDRESS` (`1.2.3.4`)
Expand Down Expand Up @@ -280,4 +286,26 @@ For a full list of modules, including the data types consumed and emitted by eac
| subdomain-hijack | 1 | Detects hijackable subdomains | subdomain_hijack |
| web-screenshots | 1 | Takes screenshots of web pages | gowitness |
<!-- END BBOT MODULE FLAGS -->
</details>

## BBOT Output Modules
BBOT can save its data to TXT, CSV, JSON, and tons of other destinations including [Neo4j](https://www.blacklanternsecurity.com/bbot/scanning/output/#neo4j), [Splunk](https://www.blacklanternsecurity.com/bbot/scanning/output/#splunk), and [Discord](https://www.blacklanternsecurity.com/bbot/scanning/output/#discord-slack-teams). For instructions on how to use these, see [Output Modules](https://www.blacklanternsecurity.com/bbot/scanning/output).

<!-- BBOT OUTPUT MODULES -->
| Module | Type | Needs API Key | Description | Flags | Consumed Events | Produced Events |
|-----------------|--------|-----------------|-----------------------------------------------------------------------------------------|----------------|--------------------------------------------------------------------------------------------------|---------------------------|
| asset_inventory | output | No | Merge hosts, open ports, technologies, findings, etc. into a single asset inventory CSV | | DNS_NAME, FINDING, HTTP_RESPONSE, IP_ADDRESS, OPEN_TCP_PORT, TECHNOLOGY, URL, VULNERABILITY, WAF | IP_ADDRESS, OPEN_TCP_PORT |
| csv | output | No | Output to CSV | | * | |
| discord | output | No | Message a Discord channel when certain events are encountered | | * | |
| emails | output | No | Output any email addresses found belonging to the target domain | email-enum | EMAIL_ADDRESS | |
| http | output | No | Send every event to a custom URL via a web request | | * | |
| human | output | No | Output to text | | * | |
| json | output | No | Output to Newline-Delimited JSON (NDJSON) | | * | |
| neo4j | output | No | Output to Neo4j | | * | |
| python | output | No | Output via Python API | | * | |
| slack | output | No | Message a Slack channel when certain events are encountered | | * | |
| splunk | output | No | Send every event to a splunk instance through HTTP Event Collector | | * | |
| subdomains | output | No | Output only resolved, in-scope subdomains | subdomain-enum | DNS_NAME, DNS_NAME_UNRESOLVED | |
| teams | output | No | Message a Teams channel when certain events are encountered | | * | |
| web_report | output | No | Create a markdown report with web assets | | FINDING, TECHNOLOGY, URL, VHOST, VULNERABILITY | |
| websocket | output | No | Output to websockets | | * | |
<!-- END BBOT OUTPUT MODULES -->
89 changes: 50 additions & 39 deletions bbot/cli.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
import asyncio
import logging
import traceback
from aioconsole import ainput
from omegaconf import OmegaConf
from contextlib import suppress
from aioconsole import stream

# fix tee buffering
sys.stdout.reconfigure(line_buffering=True)
Expand All @@ -20,6 +20,7 @@
from bbot import __version__
from bbot.modules import module_loader
from bbot.core.configurator.args import parser
from bbot.core.helpers.misc import smart_decode
from bbot.core.helpers.logger import log_to_stderr
from bbot.core.configurator import ensure_config_files, check_cli_args, environ

Expand Down Expand Up @@ -301,46 +302,56 @@ async def _main():

if not options.dry_run:
log.trace(f"Command: {' '.join(sys.argv)}")
if not options.agent_mode and not options.yes and sys.stdin.isatty():
log.hugesuccess(f"Scan ready. Press enter to execute {scanner.name}")
input()

def handle_keyboard_input(keyboard_input):
kill_regex = re.compile(r"kill (?P<module>[a-z0-9_]+)")
if keyboard_input:
log.verbose(f'Got keyboard input: "{keyboard_input}"')
kill_match = kill_regex.match(keyboard_input)
if kill_match:
module = kill_match.group("module")
if module in scanner.modules:
log.hugewarning(f'Killing module: "{module}"')
scanner.manager.kill_module(module, message="killed by user")
else:
log.warning(f'Invalid module: "{module}"')
else:
toggle_log_level(logger=log)
scanner.manager.modules_status(_log=True)

async def akeyboard_listen():
allowed_errors = 10
while 1:
keyboard_input = "a"
if sys.stdin.isatty():
if not options.agent_mode and not options.yes:
log.hugesuccess(f"Scan ready. Press enter to execute {scanner.name}")
input()

def handle_keyboard_input(keyboard_input):
kill_regex = re.compile(r"kill (?P<module>[a-z0-9_]+)")
if keyboard_input:
log.verbose(f'Got keyboard input: "{keyboard_input}"')
kill_match = kill_regex.match(keyboard_input)
if kill_match:
module = kill_match.group("module")
if module in scanner.modules:
log.hugewarning(f'Killing module: "{module}"')
scanner.manager.kill_module(module, message="killed by user")
else:
log.warning(f'Invalid module: "{module}"')
else:
toggle_log_level(logger=log)
scanner.manager.modules_status(_log=True)

# Reader
reader = stream.StandardStreamReader()
protocol = stream.StandardStreamReaderProtocol(reader)
await asyncio.get_event_loop().connect_read_pipe(lambda: protocol, sys.stdin)

async def akeyboard_listen():
try:
keyboard_input = await ainput()
except Exception:
allowed_errors -= 1
handle_keyboard_input(keyboard_input)
if allowed_errors <= 0:
break

try:
keyboard_listen_task = asyncio.create_task(akeyboard_listen())

await scanner.async_start_without_generator()
finally:
keyboard_listen_task.cancel()
with suppress(asyncio.CancelledError):
await keyboard_listen_task
allowed_errors = 10
while 1:
keyboard_input = None
try:
keyboard_input = smart_decode((await reader.readline()).strip())
allowed_errors = 10
except Exception as e:
log_to_stderr(f"Error in keyboard listen loop: {e}", level="TRACE")
log_to_stderr(traceback.format_exc(), level="TRACE")
allowed_errors -= 1
if keyboard_input is not None:
handle_keyboard_input(keyboard_input)
if allowed_errors <= 0:
break
except Exception as e:
log_to_stderr(f"Error in keyboard listen task: {e}", level="ERROR")
log_to_stderr(traceback.format_exc(), level="TRACE")

asyncio.create_task(akeyboard_listen())

await scanner.async_start_without_generator()

except bbot.core.errors.ScanError as e:
log_to_stderr(str(e), level="ERROR")
Expand Down
22 changes: 16 additions & 6 deletions bbot/core/event/base.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -565,7 +565,7 @@ def __contains__(self, other):
return host_in_host(other.host, self.host)
return False

def json(self, mode="json"):
def json(self, mode="json", siem_friendly=False):
"""
Serializes the event object to a JSON-compatible dictionary.
Expand All @@ -574,6 +574,7 @@ def json(self, mode="json"):
Parameters:
mode (str): Specifies the data serialization mode. Default is "json". Other options include "graph", "human", and "id".
siem_friendly (bool): Whether to format the JSON in a way that's friendly to SIEM ingestion by Elastic, Splunk, etc. This ensures the value of "data" is always the same type (a dictionary).
Returns:
dict: JSON-serializable dictionary representation of the event object.
Expand All @@ -585,9 +586,13 @@ def json(self, mode="json"):
j.update({i: v})
data_attr = getattr(self, f"data_{mode}", None)
if data_attr is not None:
j["data"] = data_attr
data = data_attr
else:
j["data"] = smart_decode(self.data)
data = smart_decode(self.data)
if siem_friendly:
j["data"] = {self.type: data}
else:
j["data"] = data
web_spider_distance = getattr(self, "web_spider_distance", None)
if web_spider_distance is not None:
j["web_spider_distance"] = web_spider_distance
Expand Down Expand Up @@ -1312,7 +1317,7 @@ def make_event(
)


def event_from_json(j):
def event_from_json(j, siem_friendly=False):
"""
Creates an event object from a JSON dictionary.
Expand All @@ -1335,14 +1340,19 @@ def event_from_json(j):
if required keys are missing. Make sure to validate the JSON input beforehand.
"""
try:
event_type = j["type"]
kwargs = {
"data": j["data"],
"event_type": j["type"],
"event_type": event_type,
"scans": j.get("scans", []),
"tags": j.get("tags", []),
"confidence": j.get("confidence", 5),
"dummy": True,
}
if siem_friendly:
data = j["data"][event_type]
else:
data = j["data"]
kwargs["data"] = data
event = make_event(**kwargs)

resolved_hosts = j.get("resolved_hosts", [])
Expand Down
11 changes: 5 additions & 6 deletions bbot/core/helpers/async_helpers.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,12 @@
import threading
from datetime import datetime
from queue import Queue, Empty
from cachetools import LRUCache
from .misc import human_timedelta
from contextlib import asynccontextmanager

log = logging.getLogger("bbot.core.helpers.async_helpers")

from .cache import CacheDict


class ShuffleQueue(asyncio.Queue):
def _put(self, item):
Expand All @@ -32,20 +31,20 @@ class NamedLock:
"""
Returns a unique asyncio.Lock() based on a provided string
Useful for preventing multiple operations from occuring on the same data in parallel
Useful for preventing multiple operations from occurring on the same data in parallel
E.g. simultaneous DNS lookups on the same hostname
"""

def __init__(self, max_size=1000):
self._cache = CacheDict(max_size=max_size)
self._cache = LRUCache(maxsize=max_size)

@asynccontextmanager
async def lock(self, name):
try:
lock = self._cache.get(name)
lock = self._cache[name]
except KeyError:
lock = _Lock(name)
self._cache.put(name, lock)
self._cache[name] = lock
async with lock:
yield

Expand Down
Loading

0 comments on commit bc57ab5

Please sign in to comment.