Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev -> Stable 2.2.0 #1919

Merged
merged 97 commits into from
Nov 18, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
39465c5
add initial dnsbimi module
colin-stubbs Sep 3, 2024
8068d4c
Merge branch 'dnsbimi' of https://github.com/colin-stubbs/bbot into d…
colin-stubbs Sep 3, 2024
3b66003
Bump mkdocs-material from 9.5.42 to 9.5.43
dependabot[bot] Nov 4, 2024
d8b2ec3
Bump pytest-cov from 5.0.0 to 6.0.0
dependabot[bot] Nov 4, 2024
3eb4135
Merge pull request #1915 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Nov 4, 2024
b2c3bb2
Merge pull request #1914 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Nov 4, 2024
d28fe33
Bump werkzeug from 3.0.4 to 3.1.1
dependabot[bot] Nov 4, 2024
3382fea
Revert "Merge pull request #1836 from domwhewell-sage/folder_crawling"
domwhewell-sage Nov 4, 2024
fcc9f79
Disable unstructured as folders will no longer be crawled
domwhewell-sage Nov 4, 2024
3c2d05b
Merge pull request #1917 from domwhewell-sage/revert_speculate_changes
TheTechromancer Nov 4, 2024
b729f28
Merge pull request #1913 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Nov 4, 2024
db3bdfd
fix ansible executable
invalid-email-address Nov 4, 2024
ba598ad
fix fedora
invalid-email-address Nov 4, 2024
02069d1
Merge pull request #1921 from blacklanternsecurity/fix-fedora
TheTechromancer Nov 4, 2024
84fdb87
Update trufflehog
blsaccess Nov 5, 2024
a083f7a
[create-pull-request] automated change
TheTechromancer Nov 5, 2024
3f3f35b
Merge pull request #1907 from blacklanternsecurity/update-trufflehog
TheTechromancer Nov 5, 2024
157fed6
fix double target printing
invalid-email-address Nov 5, 2024
812a0d7
Merge pull request #1911 from blacklanternsecurity/update-docs
TheTechromancer Nov 5, 2024
e0ea2cb
Update trufflehog
blsaccess Nov 6, 2024
05875e3
Merge pull request #1925 from blacklanternsecurity/update-trufflehog
TheTechromancer Nov 6, 2024
442daa6
fix
invalid-email-address Nov 6, 2024
72a250a
Merge pull request #1924 from blacklanternsecurity/fix-target-printing
TheTechromancer Nov 6, 2024
e8ec678
unstructured -> extractous
invalid-email-address Nov 6, 2024
1b433d0
oops
invalid-email-address Nov 6, 2024
7bc6a43
Update trufflehog
blsaccess Nov 7, 2024
edd2d16
[create-pull-request] automated change
TheTechromancer Nov 7, 2024
8b9f03a
blacked
invalid-email-address Nov 7, 2024
8db576f
add native filetype + compression detection
invalid-email-address Nov 7, 2024
5f5349e
magic
invalid-email-address Nov 7, 2024
1cb32e2
fix tests
invalid-email-address Nov 7, 2024
8100118
add libgl
invalid-email-address Nov 7, 2024
27258a1
Merge pull request #1930 from blacklanternsecurity/fix-ubuntu-chrome
TheTechromancer Nov 7, 2024
17a55eb
Merge pull request #1929 from blacklanternsecurity/magic
TheTechromancer Nov 7, 2024
3aaccda
Merge pull request #1927 from blacklanternsecurity/extractous
TheTechromancer Nov 7, 2024
475ee5d
Merge pull request #1928 from blacklanternsecurity/update-trufflehog
TheTechromancer Nov 7, 2024
0f1db8c
Merge pull request #1926 from blacklanternsecurity/update-docs
TheTechromancer Nov 7, 2024
f0eca9a
Added new module jadx
domwhewell-sage Nov 7, 2024
4f3b518
Add safe flag
domwhewell-sage Nov 7, 2024
2e0c1f2
add sqlite output module
invalid-email-address Nov 7, 2024
e344054
update docs
invalid-email-address Nov 7, 2024
770bb52
more docs
invalid-email-address Nov 7, 2024
4a07535
more docs
invalid-email-address Nov 7, 2024
a0f6e97
Merge pull request #1933 from blacklanternsecurity/sql-output-module
TheTechromancer Nov 8, 2024
7ef7461
version bump badsecrets/baddns
liquidsec Nov 8, 2024
65ac448
Merge pull request #1939 from blacklanternsecurity/dependency-bump-ba…
liquidsec Nov 8, 2024
3e67a9f
Added test to check the detected file type
domwhewell-sage Nov 9, 2024
9c512ad
fix ubuntu chrome
invalid-email-address Nov 9, 2024
455051d
Made changes to apkpure to create the file extension based on the Con…
domwhewell-sage Nov 9, 2024
64ddb45
Change fedora package
domwhewell-sage Nov 9, 2024
6aa2a5c
Change to regex to get the extension
domwhewell-sage Nov 9, 2024
5a98cf3
set the JAVA_HOME environment variable on fedora
domwhewell-sage Nov 9, 2024
ed03047
[create-pull-request] automated change
TheTechromancer Nov 10, 2024
0173b87
Please set the environment variable on fedora
domwhewell-sage Nov 10, 2024
4d69f3c
lint
domwhewell-sage Nov 10, 2024
82c549e
Merge pull request #1934 from blacklanternsecurity/update-docs
TheTechromancer Nov 10, 2024
41e8c29
Set the java home in the jadx script
domwhewell-sage Nov 10, 2024
7a42e11
Update trufflehog
blsaccess Nov 11, 2024
13652c3
Bump mkdocstrings from 0.26.2 to 0.27.0
dependabot[bot] Nov 11, 2024
8465e21
Bump regex from 2024.9.11 to 2024.11.6
dependabot[bot] Nov 11, 2024
454235e
Bump werkzeug from 3.1.1 to 3.1.3
dependabot[bot] Nov 11, 2024
91eab76
Bump tldextract from 5.1.2 to 5.1.3
dependabot[bot] Nov 11, 2024
58624cf
Bump mkdocs-material from 9.5.43 to 9.5.44
dependabot[bot] Nov 11, 2024
3e2a738
Merge pull request #1940 from blacklanternsecurity/update-trufflehog
TheTechromancer Nov 11, 2024
e952280
Merge pull request #1952 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Nov 11, 2024
a591691
Merge pull request #1951 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Nov 11, 2024
fc0d877
Merge pull request #1950 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Nov 11, 2024
be092b8
Ensure its inserted after the shebang
domwhewell-sage Nov 11, 2024
31c18b9
Merge branch 'dev' into jadx_module
domwhewell-sage Nov 11, 2024
c0be101
Dont have to set JAVA_HOME can just install which to allow jadx start…
domwhewell-sage Nov 11, 2024
328de60
fix ubuntu again?
invalid-email-address Nov 11, 2024
cf7686b
test gowitness only
invalid-email-address Nov 11, 2024
1643946
Merge pull request #1949 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Nov 11, 2024
ee76f68
Merge pull request #1948 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Nov 11, 2024
836d115
only gowitness
invalid-email-address Nov 11, 2024
437ac86
wat
invalid-email-address Nov 11, 2024
5de9452
Merge pull request #1941 from blacklanternsecurity/fix-ubuntu-chrome
TheTechromancer Nov 11, 2024
41ac684
Merge branch 'dev' into jadx_module
domwhewell-sage Nov 12, 2024
154c354
Merge pull request #1932 from domwhewell-sage/jadx_module
TheTechromancer Nov 13, 2024
4c2440a
make java a shared dep
invalid-email-address Nov 13, 2024
0078a3a
jadx debugging
invalid-email-address Nov 13, 2024
9d8d7a1
fix java dep
invalid-email-address Nov 13, 2024
09f7ec3
Merge pull request #1956 from blacklanternsecurity/fix-jadx
TheTechromancer Nov 13, 2024
598d9f3
[create-pull-request] automated change
TheTechromancer Nov 14, 2024
0349750
Merge pull request #1946 from blacklanternsecurity/update-docs
TheTechromancer Nov 14, 2024
ef887da
Merge remote-tracking branch 'colin-stubbs/stable' into dnsbimi
colin-stubbs Nov 15, 2024
be3b685
update for PR
colin-stubbs Nov 15, 2024
796d633
Adjust based no review by Techromanacer
colin-stubbs Nov 16, 2024
be2a005
Merge pull request #1965 from colin-stubbs/dnsbimi
TheTechromancer Nov 16, 2024
5ab90fa
Update trufflehog
blsaccess Nov 17, 2024
24fb696
Merge pull request #1968 from blacklanternsecurity/update-trufflehog
TheTechromancer Nov 17, 2024
39796f0
[create-pull-request] automated change
TheTechromancer Nov 17, 2024
a55e9dd
Merge pull request #1964 from blacklanternsecurity/update-docs
TheTechromancer Nov 17, 2024
a54199b
Use Safe Defaults for `lxml` Parsers (#1)
pixeebot[bot] Nov 17, 2024
b1d8b94
Merge pull request #1973 from Pixeebot-2-0/stable
TheTechromancer Nov 17, 2024
b14bad5
update to match new extractous API changes
invalid-email-address Nov 18, 2024
3945fd1
Merge pull request #1976 from blacklanternsecurity/fix-extractous
TheTechromancer Nov 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 2 additions & 23 deletions bbot/modules/internal/speculate.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import random
import ipaddress
from pathlib import Path

from bbot.core.helpers import validators
from bbot.modules.internal.base import BaseInternalModule
Expand All @@ -24,21 +23,19 @@ class speculate(BaseInternalModule):
"SOCIAL",
"AZURE_TENANT",
"USERNAME",
"FILESYSTEM",
]
produced_events = ["DNS_NAME", "OPEN_TCP_PORT", "IP_ADDRESS", "FINDING", "ORG_STUB", "FILESYSTEM"]
produced_events = ["DNS_NAME", "OPEN_TCP_PORT", "IP_ADDRESS", "FINDING", "ORG_STUB"]
flags = ["passive"]
meta = {
"description": "Derive certain event types from others by common sense",
"created_date": "2022-05-03",
"author": "@liquidsec",
}

options = {"max_hosts": 65536, "ports": "80,443", "ignore_folders": [".git"]}
options = {"max_hosts": 65536, "ports": "80,443"}
options_desc = {
"max_hosts": "Max number of IP_RANGE hosts to convert into IP_ADDRESS events",
"ports": "The set of ports to speculate on",
"ignore_folders": "Subfolders to ignore when crawling downloaded folders",
}
scope_distance_modifier = 1
_priority = 4
Expand Down Expand Up @@ -75,13 +72,6 @@ async def setup(self):
self.hugewarning(f'Enabling the "portscan" module is highly recommended')
self.range_to_ip = False

self.ignored_folders = self.config.get("ignore_folders", [])

return True

async def filter_event(self, event):
if event.type == "FILESYSTEM" and "folder" not in event.tags:
return False, "Event is not a folder"
return True

async def handle_event(self, event):
Expand Down Expand Up @@ -205,14 +195,3 @@ async def handle_event(self, event):
email_event = self.make_event(email, "EMAIL_ADDRESS", parent=event, tags=["affiliate"])
if email_event:
await self.emit_event(email_event, context="detected {event.type}: {event.data}")

# FILESYSTEM (folder) --> FILESYSTEM (files)
if event.type == "FILESYSTEM":
folder_path = Path(event.data["path"])
for file_path in folder_path.rglob("*"):
# If the file is not in an ignored folder and if it has an allowed extension raise it as a FILESYSTEM event
if not any(ignored_folder in str(file_path) for ignored_folder in self.ignored_folders):
file_event = self.make_event(
{"path": str(file_path)}, "FILESYSTEM", tags=["parsed_folder", "file"], parent=event
)
await self.emit_event(file_event)
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ class TestGit_CloneWithBlob(TestGit_Clone):
config_overrides = {"folder_blobs": True}

def check(self, module_test, events):
filesystem_events = [e for e in events if e.type == "FILESYSTEM" and "folder" in e.tags]
filesystem_events = [e for e in events if e.type == "FILESYSTEM"]
assert len(filesystem_events) == 1
assert all(["blob" in e.data for e in filesystem_events])
filesystem_event = filesystem_events[0]
Expand Down
Loading