BBOT 2.0

[TheTechromancer]

This is the main PR for BBOT 2.0 which will be merged into dev at its release.

Beginning with #1058, features destined for BBOT 2.0 will be merged into this branch.

New Features

• Presets (-p / --presets)
• New discovery_context attribute on every event which contains a description of exactly how the event was discovered.
  • Also, a new discovery_path attribute that contains the complete chain of parent contexts all the way back to the scan target.
• New --json command-line flag: alias for -c modules.stdout.format=json
• New --brief command-line flag for cleaner output
• Importing Scanner is now easier:

from bbot import Scanner

• New dns.brute helper:

async def handle_event(self, event):
    query = self.make_query(event)
    self.info(f"Brute-forcing subdomains for {query} (source: {event.data})")
    for hostname in await self.helpers.dns.brute(self, query, self.subdomain_list):
        await self.emit_event(hostname, "DNS_NAME", source=event)

• New request_batch helper:

async for url, response in self.helpers.request_batch(urls):
    ...

• New async regex helper (you only need to use this if your regex is slow or you're searching a lot of text, i.e. a web response):

r = self.helpers.re.compile('\d+')
matches = await self.helpers.re.findall(r, text)

Breaking Changes

• --current-config is now --current-preset (or --current-preset-full)
• --help-all has gone away. To list module options, you now do --list-module-options
• Massdns threads are now a global option:

# before
bbot -c modules.massdns.max_resolvers=5000

# after
bbot -c dns.brute_threads=5000

• DNS, web, and scope settings are now nested:

### SCOPE ###

scope:
  # Filter by scope distance which events are displayed in the output
  # 0 == show only in-scope events (affiliates are always shown)
  # 1 == show all events up to distance-1 (1 hop from target)
  report_distance: 0
  # How far out from the main scope to search
  # Do not change this setting unless you know what you're doing
  search_distance: 0

### DNS ###

dns:
  # Completely disable DNS resolution (careful if you have IP whitelists/blacklists, consider using minimal=true instead)
  disable: false
  # Speed up scan by not creating any new DNS events, and only resolving A and AAAA records
  minimal: false
  # How many instances of the dns module to run concurrently
  threads: 20
  # How many concurrent DNS resolvers to use when brute-forcing
  # (under the hood this is passed through directly to massdns -s)
  brute_threads: 1000
  # How far away from the main target to explore via DNS resolution (independent of scope.search_distance)
  # This is safe to change
  search_distance: 1
  # Limit how many DNS records can be followed in a row (stop malicious/runaway DNS records)
  runaway_limit: 5
  # DNS query timeout
  timeout: 5
  # How many times to retry DNS queries
  retries: 1
  # Completely disable BBOT's DNS wildcard detection
  wildcard_disable: False
  # Disable BBOT's DNS wildcard detection for select domains
  wildcard_ignore: []
  # How many sanity checks to make when verifying wildcard DNS
  # Increase this value if BBOT's wildcard detection isn't working
  wildcard_tests: 10
  # Skip DNS requests for a certain domain and rdtype after encountering this many timeouts or SERVFAILs
  # This helps prevent faulty DNS servers from hanging up the scan
  abort_threshold: 50
  # Don't show PTR records containing IP addresses
  filter_ptrs: true
  # Enable/disable debug messages for DNS queries
  debug: false
  # For performance reasons, always skip these DNS queries
  # Microsoft's DNS infrastructure is misconfigured so that certain queries to mail.protection.outlook.com always time out
  omit_queries:
    - SRV:mail.protection.outlook.com
    - CNAME:mail.protection.outlook.com
    - TXT:mail.protection.outlook.com

### WEB ###

web:
  # HTTP proxy
  http_proxy: 
  # Web user-agent
  user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.2151.97
  # Set the maximum number of HTTP links that can be followed in a row (0 == no spidering allowed)
  spider_distance: 0
  # Set the maximum directory depth for the web spider
  spider_depth: 1
  # Set the maximum number of links that can be followed per page
  spider_links_per_page: 25
  # HTTP timeout (for Python requests; API calls, etc.)
  http_timeout: 10
  # HTTP timeout (for httpx)
  httpx_timeout: 5
  # Custom HTTP headers (e.g. cookies, etc.)
  # in the format { "Header-Key": "header_value" }
  # These are attached to all in-scope HTTP requests
  # Note that some modules (e.g. github) may end up sending these to out-of-scope resources
  http_headers: {}
  # HTTP retries (for Python requests; API calls, etc.)
  http_retries: 1
  # HTTP retries (for httpx)
  httpx_retries: 1
  # Enable/disable debug messages for web requests/responses
  debug: false
  # Maximum number of HTTP redirects to follow
  http_max_redirects: 5
  # Whether to verify SSL certificates
  ssl_verify: false

# Tool dependencies
deps:
  ffuf:
    version: "2.1.0"

• All module configs are now directly under modules (output_modules and internal_modules have gone away):

# before
bbot -c output_modules.splunk.url=http://example.com

# after
bbot -c modules.splunk.url=http://example.com

• Event source attribute has been renamed to parent
• max_event_handlers has been renamed to module_threads

Merged Features:

• Presets #1058
• DNS Engine - Offload DNS to Dedicated Process #1231
• Intercept Modules #1242
• Shared Ansible Dependencies #1270
• Faster Regexes #1278
• Implement Radix Tree #1295
• Helperify Massdns #1303
• Stdout Output Module #1234
• Fix omitted event types #1331
• Shorten ingress queue #1332
• Clean up debug messages #1333
• Improve --brief stdout #1334
• HTTP Engine - Offload Web Requests to Dedicated Process #1340
• Ensure BBOT config file gets created on startup #1350
• better pickle/unpickle error handling #1351
• Misc Minor BBOT 2.0 Tweaks #1358
• Paramminer Extract Validation / Tests #1359
• Include "host" as attribute in event JSON #1364
• Fix whitelisting scope bug #1366
• Adding custom headers CLI argument / tests #1368
• Fix subdomain enum duplicate query bug #1373
• Portscanner Rework #1375
• Regex Performance Improvements, Tests #1393
• Module Metadata: Author Name + Created Date #1395
• Event Context #1383
• Config Cleanup and BBOT 2.0 Documentation #1419
• Restore secrets.yml functionality #1479
• Nuclei - support for custom headers #1482
• Option to limit the number of table lines logged to the console #1481
• Preset cache #1339
• Make Target Events Config-Aware #1496
• Fix Minor Context Bug #1499
• Limit dnscommonsrv depth #1507
• Fix bug with ansible path #1509
• Fix portscan dependency bug #1510
• Replace excavate with YARA, and introduce web_parameters #1471
• Module killing/status tweaks #1530
• Add RAW_DNS_RECORD Event Type #1532
• Rename dns --> dnsresolve, cloud --> cloudcheck #1535
• Convert bucket modules to use request_batch #1536
• BBOT 2.0 Prerelease CLI Warning #1541

Feature Checklist:

• Presets #1058
• DNS Engine - Offload DNS to Dedicated Process #1231
• Convert bucket modules to use request_batch #1536
• Better error handling in multiprocess pool #1297
• Subdomain enum bug w/ multiple queries:
• New Excavate Module: Implement Yara #1252
• Only tag DNS events with DNS-related tags (cname-record, cname-wildcard, etc.)
  • Include DNS children in json output
• Offload HTTP work into dedicated process #1223
• Perf optimization: don't create dummy events in target.py
• Event Discovery Context #1352
• Fix Error in keyboard listen task: 'bool' object has no attribute '_queue'
• Tests for custom target types, e.g. ORG:evilcorp
• Tests for url_extension_blacklist and url_extension_httpx_only
• Don't shuffle output module queues
• Clean up spider-danger / URL_UNVERIFIED situation (URL_UNVERIFIEDs shouldn't be deduped by spider-danger #1208) - most likely will depend on HTTP engine rework / httpx replacement
• Clean up console output to make tool friendlier to newcomers (see https://twitter.com/h4x0r_dz/status/1781752033424367690)
• Rename engine processes
• Implement radix tree cloudcheck#15
• Preset cache
• Stdout Output Module #1234
• Intercept Modules #1242
• Shared Ansible Dependencies #1270
• Graduate Agent to Standalone Tool #1195
  • remove agent-specific logic from bbot core
  • move it into its own repo
• Portscanner Rework #913
• Consider using AnyIO #1080
• Massdns Rework #1255
• Benchmark uvloop #942
• Asyncify regexes #1184
• Usability / code readability improvements:
  • Rename event.source to be event.parent. event.source is confusing.
  • Presets: Rename output.ndjson back to .json #1227
• Author, created date for modules
  • Docs table
  • Tests
• Slick "how it works" graphic for README
  • Simple graphic showing the circular flow of events in BBOT:
    • DNS resolution, whitelisting/blacklisting, excavate/speculate, etc.
  • In docs: Detailed graph showing all watched/produced interactions between modules (autogenerate?)
• Make a review pass on documentation
  • Full start-to-finish tutorial on how to write a module (including writing/debugging tests)

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 87.96814% with 559 lines in your changes missing coverage. Please review.

Project coverage is 93%. Comparing base (a75f846) to head (482bba8).

Files	Patch %	Lines
bbot/core/helpers/dns/engine.py	77%	78 Missing ⚠️
bbot/modules/internal/excavate.py	87%	67 Missing ⚠️
bbot/cli.py	65%	59 Missing ⚠️
bbot/core/engine.py	80%	58 Missing ⚠️
bbot/core/helpers/web/engine.py	79%	39 Missing ⚠️
bbot/modules/base.py	81%	29 Missing ⚠️
bbot/modules/portscan.py	87%	28 Missing ⚠️
bbot/core/modules.py	90%	24 Missing ⚠️
bbot/core/config/logger.py	90%	15 Missing ⚠️
bbot/modules/paramminer_headers.py	70%	15 Missing ⚠️
... and 44 more

Additional details and impacted files

@@           Coverage Diff           @@
##             dev   #1235     +/-   ##
=======================================
+ Coverage     92%     93%     +1%     
=======================================
  Files        327     347     +20     
  Lines      21117   25221   +4104     
=======================================
+ Hits       19381   23215   +3834     
- Misses      1736    2006    +270     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.