Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BBOT 2.0 #1235

Merged
merged 782 commits into from
Jul 8, 2024
Merged

BBOT 2.0 #1235

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
782 commits
Select commit Hold shift + click to select a range
02ca9ba
fixing relative path parsing
liquidsec Jun 13, 2024
089b53d
merge dev
TheTechromancer Jun 14, 2024
31a80ae
blacked
TheTechromancer Jun 14, 2024
038500a
remove massdns
TheTechromancer Jun 14, 2024
58f071a
added custom yara rules initial
liquidsec Jun 14, 2024
dfd770a
custom arg for yara rules
liquidsec Jun 14, 2024
dad4dd9
merge dev
TheTechromancer Jun 17, 2024
b3726d8
adding jquery parameter extraction
liquidsec Jun 17, 2024
77ce3cd
more excavate functionality, starting to fix tests
liquidsec Jun 17, 2024
c0c2f38
adding non-http scheme excavate submodule
liquidsec Jun 18, 2024
b229d45
don't shuffle portscan's incoming queue
TheTechromancer Jun 18, 2024
b8121a7
minor bug fixes, improving non-http scheme detection, tests
liquidsec Jun 18, 2024
1d497f1
misc cleanup
liquidsec Jun 18, 2024
95ab483
small cleanup
liquidsec Jun 18, 2024
2daedb6
refactor to avoid variable scope issues
liquidsec Jun 18, 2024
4b72c44
major refactor, in the midst of spider-danger surgery
liquidsec Jun 18, 2024
70f117c
a little cleanup
liquidsec Jun 18, 2024
25e5a69
Merge branch 'bbot-2.0' into new-presets
liquidsec Jun 18, 2024
f7b91a5
Little updates
liquidsec Jun 19, 2024
c46da92
merge dev
TheTechromancer Jun 19, 2024
065a560
restore secrets.yml functionality
TheTechromancer Jun 20, 2024
3528599
fix gowitness tests
TheTechromancer Jun 20, 2024
c7c9d13
fix tests
TheTechromancer Jun 20, 2024
ed37bc2
fix wpscan tests
TheTechromancer Jun 20, 2024
3003eaf
limit table lines
TheTechromancer Jun 20, 2024
1e5ff17
Merge pull request #1448 from blacklanternsecurity/bbot-2.0-dev-merge
TheTechromancer Jun 20, 2024
26992f8
modules oopsie safeguard, include secrets.yml in config
TheTechromancer Jun 20, 2024
6612687
merge 2.0
TheTechromancer Jun 20, 2024
e00724b
fix undefined variable
TheTechromancer Jun 20, 2024
ad474f6
nuclei custom header support
TheTechromancer Jun 20, 2024
999f075
merge 2.0
TheTechromancer Jun 20, 2024
ab51947
first yara test
liquidsec Jun 20, 2024
dfa6258
Merge branch 'bbot-2.0' into new-presets
liquidsec Jun 20, 2024
776c916
Merge pull request #1482 from blacklanternsecurity/nuclei-header-support
TheTechromancer Jun 20, 2024
98739e5
Merge pull request #1481 from blacklanternsecurity/limit-table-lines
TheTechromancer Jun 20, 2024
105219a
fix ansible debug
TheTechromancer Jun 20, 2024
fd09473
stuff
TheTechromancer Jun 20, 2024
8a838ec
Merge pull request #1472 from blacklanternsecurity/new-presets
TheTechromancer Jun 20, 2024
4efc7b0
fix tests
TheTechromancer Jun 20, 2024
fcc549d
merge bbot 2.0
TheTechromancer Jun 20, 2024
cec7b10
Merge pull request #1479 from blacklanternsecurity/fix-missing-config
TheTechromancer Jun 20, 2024
d9d4567
Merge pull request #1339 from blacklanternsecurity/preset-cache
TheTechromancer Jun 20, 2024
a3ab3d9
major refactor around spider danger
liquidsec Jun 21, 2024
cc64908
fixing max links test to fit new system
liquidsec Jun 21, 2024
b47e4dd
bug squashing
liquidsec Jun 21, 2024
840bdc2
only publish docs to stable
TheTechromancer Jun 21, 2024
50973a2
merge dev
TheTechromancer Jun 21, 2024
e7515f8
fix logging import
TheTechromancer Jun 21, 2024
a844de4
delete dead code
TheTechromancer Jun 21, 2024
f074a3c
preset docs
TheTechromancer Jun 24, 2024
f3447b1
querystring tests
liquidsec Jun 24, 2024
ef30963
defer target event creation until preset.bake(), add target hashing f…
TheTechromancer Jun 24, 2024
3e8f4a6
merge from bake-targets
liquidsec Jun 24, 2024
a6ae7e5
fix tests
TheTechromancer Jun 24, 2024
88bd956
fix tests
TheTechromancer Jun 24, 2024
c2cc0b5
fix json tests
TheTechromancer Jun 24, 2024
66e3724
major revamp paramminer
liquidsec Jun 25, 2024
d1e37c3
remove defunct is_spider_danger
liquidsec Jun 25, 2024
e977f49
merge with bake-targets
liquidsec Jun 25, 2024
4dfbf4f
fix tests
TheTechromancer Jun 25, 2024
ba64842
Merge branch 'bake-targets' into spider-dangerous
liquidsec Jun 25, 2024
fe4609b
fix tests again
TheTechromancer Jun 25, 2024
ee0b86b
fix context issue
TheTechromancer Jun 25, 2024
bd35f9d
hash by data_id
TheTechromancer Jun 25, 2024
5200881
lots of bug fixes / test updates
liquidsec Jun 25, 2024
e1c1307
Merge remote-tracking branch 'origin/bake-targets' into spider-dangerous
liquidsec Jun 25, 2024
70fcceb
fixing paramminer getparams tests
liquidsec Jun 25, 2024
e8c2a05
fixing paramminer tests
liquidsec Jun 25, 2024
6d97866
paramminer handle speculative parameters
liquidsec Jun 25, 2024
1e70c0c
Merge pull request #1496 from blacklanternsecurity/bake-targets
TheTechromancer Jun 25, 2024
993f9eb
removing scraped web_parameter dynamic stuff
liquidsec Jun 25, 2024
c017ad0
Merge branch 'bbot-2.0' into spider-dangerous
liquidsec Jun 25, 2024
52a5f35
Merge pull request #1499 from blacklanternsecurity/context-tweaks
TheTechromancer Jun 25, 2024
c95068d
more tests / small bug fixes
liquidsec Jun 25, 2024
3822eeb
oops
liquidsec Jun 25, 2024
77845a6
remove debug junk
liquidsec Jun 25, 2024
25e86a1
add target_hash to scan
TheTechromancer Jun 25, 2024
27d1b3c
test tweaks and bug fixes agian
liquidsec Jun 25, 2024
009b95b
fixing bug where parameter extration wouldn't disable
liquidsec Jun 25, 2024
39603da
black
liquidsec Jun 25, 2024
b2e2cb3
flake
liquidsec Jun 25, 2024
9b9e91a
flake again
liquidsec Jun 25, 2024
3c36dfd
Merge pull request #1486 from blacklanternsecurity/spider-dangerous
liquidsec Jun 25, 2024
d83ceb0
updating poetry.lock
liquidsec Jun 25, 2024
2bce9d3
updating querystring remove test
liquidsec Jun 25, 2024
2e01350
turning off references by default
liquidsec Jun 26, 2024
5e67649
We have found a witch, may we burn her?
liquidsec Jun 26, 2024
0f014a6
one last gremlin
liquidsec Jun 26, 2024
5702f5b
Merge pull request #1502 from blacklanternsecurity/baddns-references-…
TheTechromancer Jun 26, 2024
51dc1dc
fixing scope tests, improving yara url regex
liquidsec Jun 26, 2024
3d6c3ba
reworking recursive decode implementation
liquidsec Jun 26, 2024
ed2f421
removing debugging statements from excavate
liquidsec Jun 26, 2024
e32cb28
better target tests, whitelist/blacklist improvements
TheTechromancer Jun 26, 2024
67666b5
fixing discord test
liquidsec Jun 26, 2024
4d7cdc9
fixing json param extract bug
liquidsec Jun 26, 2024
c5fb632
steady work on bbot-io compatibility
TheTechromancer Jun 26, 2024
f5330ee
fixing dupes with yara results
liquidsec Jun 26, 2024
0544eb5
temporary fix for yara rule dedupe issues
liquidsec Jun 26, 2024
8eedabd
removing debugging
liquidsec Jun 26, 2024
526d7ee
parameter extraction tests require WEB_PARAMETER ingesting module
liquidsec Jun 26, 2024
1ce2332
Merge pull request #1393 from blacklanternsecurity/regex-tests
TheTechromancer Jun 27, 2024
4113e04
limit dnscommonsrv depth
TheTechromancer Jun 27, 2024
2d4a1a3
Merge pull request #1507 from blacklanternsecurity/dnscommonsrv-speedup
TheTechromancer Jun 27, 2024
540dd0e
They spent all of their time asking if they could, and they never sto…
liquidsec Jun 27, 2024
0213084
more event init order fixes
liquidsec Jun 27, 2024
e913a0f
paramminer finish sort results / fix tests
liquidsec Jun 27, 2024
0055c01
fixing slack test
liquidsec Jun 27, 2024
08a6400
fixing teams test
liquidsec Jun 27, 2024
f749c0d
fixing sleeper-cell bug
liquidsec Jun 25, 2024
48cc231
Merge branch 'bbot-2.0' into yara-excavate
liquidsec Jun 27, 2024
55e3d3c
fix ansible path issue
TheTechromancer Jun 27, 2024
6c13c80
fixing but with failed event creation
liquidsec Jun 27, 2024
7c7196f
Merge pull request #1509 from blacklanternsecurity/fix-ansible-bug
TheTechromancer Jun 27, 2024
5f03e18
splitting out dns_regex functionality for yara vs general purpose
liquidsec Jun 27, 2024
29bb2d6
fix portscan bug
TheTechromancer Jun 27, 2024
5092b19
remove debug statement
TheTechromancer Jun 27, 2024
f241909
Merge pull request #1510 from blacklanternsecurity/fix-ansible-bug
TheTechromancer Jun 27, 2024
ae0b38d
fixing debug msg
liquidsec Jun 27, 2024
aabf5b2
adding better debugging around yara compiling
liquidsec Jun 27, 2024
f62e8d4
Merge branch 'bbot-2.0' into yara-excavate
liquidsec Jun 27, 2024
20581cd
separating yara and python regexes further. They just can't get along :(
liquidsec Jun 27, 2024
977bbb5
yara docs initial
liquidsec Jun 27, 2024
0a09480
Merge branch 'bbot-2.0' into yara-documentation
liquidsec Jun 27, 2024
3e812c9
fix double printing of bbot logo
TheTechromancer Jun 27, 2024
86f8ce2
Merge branch 'yara-excavate' into yara-documentation
liquidsec Jun 27, 2024
c6b462d
Merge pull request #1514 from blacklanternsecurity/fix-ansible-bug
TheTechromancer Jun 27, 2024
6a5ba46
Merge branch 'bbot-2.0' into yara-excavate
liquidsec Jun 28, 2024
12c9e24
tweak to custom yara messaging
liquidsec Jun 28, 2024
c1fee11
Merge branch 'yara-excavate' into yara-documentation
liquidsec Jun 28, 2024
5a0f374
yara docs
liquidsec Jun 28, 2024
055d152
edits
liquidsec Jun 28, 2024
98aee52
fix scan tests
TheTechromancer Jun 28, 2024
9aa61d2
Update custom_yara_rules.md
TheTechromancer Jun 28, 2024
c284bd3
Merge pull request #1513 from blacklanternsecurity/yara-documentation
TheTechromancer Jun 28, 2024
60b48da
portscan improvements
TheTechromancer Jun 28, 2024
7cf416e
allow multiple events to be correlated per IP
TheTechromancer Jun 28, 2024
20ff230
fix json tests
TheTechromancer Jul 1, 2024
bfec228
cleaner shutdown of engines
TheTechromancer Jul 1, 2024
04e81a2
Merge pull request #1504 from blacklanternsecurity/bbot-io-testing
TheTechromancer Jul 1, 2024
6db46f0
Merge branch 'bbot-2.0' into portscan-fix
TheTechromancer Jul 1, 2024
69e649f
better tests for portscan
TheTechromancer Jul 1, 2024
7ea21f2
better tests for portscan module
TheTechromancer Jul 1, 2024
1f93b50
fixed wafw00f bug, wrote test for it
TheTechromancer Jul 1, 2024
64c8ebc
Merge pull request #1517 from blacklanternsecurity/portscan-fix
TheTechromancer Jul 1, 2024
946b7fc
small engine tweak
TheTechromancer Jul 1, 2024
0bf4ea4
pytest things
TheTechromancer Jul 1, 2024
b6a64e6
engine tweaks
TheTechromancer Jul 1, 2024
8276149
Merge pull request #1520 from blacklanternsecurity/wafw00f-fix
TheTechromancer Jul 1, 2024
016fe52
pytest tweaks
TheTechromancer Jul 1, 2024
b6b920e
fix conflicts
TheTechromancer Jul 1, 2024
873632b
set thread title
TheTechromancer Jul 2, 2024
ad7dae2
close context
TheTechromancer Jul 2, 2024
dcc714e
destroyyyyy
TheTechromancer Jul 2, 2024
aa05374
add libpcap to masscan deps
TheTechromancer Jul 2, 2024
fd2d64d
fix package name
TheTechromancer Jul 2, 2024
bddae7c
switch to -dev
TheTechromancer Jul 2, 2024
2e13693
fix masscan top ports issue
TheTechromancer Jul 2, 2024
a254acf
Merge pull request #1521 from blacklanternsecurity/bbot-io-testing
TheTechromancer Jul 2, 2024
4e7fd05
add top ports
TheTechromancer Jul 2, 2024
a5b13ad
merge dev
TheTechromancer Jul 2, 2024
28ffa9f
merge 2.0
TheTechromancer Jul 2, 2024
055d1c6
merge killing multiple modules
TheTechromancer Jul 2, 2024
7987a4e
Merge branch 'bbot-2.0' into yara-excavate
liquidsec Jul 2, 2024
013988f
fixing handling of redirects/location headers
liquidsec Jul 3, 2024
9e1a5c8
validator docstring
TheTechromancer Jul 3, 2024
01997b8
merge 2.0
TheTechromancer Jul 3, 2024
1a4c875
blacked
TheTechromancer Jul 3, 2024
71d5484
fix cli tests
TheTechromancer Jul 3, 2024
612dd1b
update readme
TheTechromancer Jul 3, 2024
a9f5a44
rework event dudupe initial
liquidsec Jul 3, 2024
eccff28
fixing test
liquidsec Jul 4, 2024
4f07415
spacing
liquidsec Jul 4, 2024
9531a63
fixing querystring_collapse dedupe, adding tests
liquidsec Jul 4, 2024
59dcb10
black
liquidsec Jul 4, 2024
808b1ad
fixing bug with URL dedupe
liquidsec Jul 4, 2024
8172db4
more fixes for URL dedupe
liquidsec Jul 4, 2024
b191770
detaching discovery_context from rule object
liquidsec Jul 4, 2024
07d0d50
removing debugging
liquidsec Jul 4, 2024
9c972ac
revising test
liquidsec Jul 4, 2024
81f60ff
black
liquidsec Jul 4, 2024
0d244b4
make WEB_PARAMETER events omitted from output by default
liquidsec Jul 4, 2024
7257d34
small tweaks
TheTechromancer Jul 4, 2024
c59d8a2
blacked
TheTechromancer Jul 4, 2024
4fa1476
Merge pull request #1528 from blacklanternsecurity/event-dedupe-changes
TheTechromancer Jul 4, 2024
9873bf2
module killing/status tweaks thanks to @amiremami
TheTechromancer Jul 4, 2024
2c54502
add speed counter
TheTechromancer Jul 4, 2024
9f993be
fix speed test
TheTechromancer Jul 4, 2024
e57af61
Merge pull request #1523 from blacklanternsecurity/bbot-2.0-dev-merge
TheTechromancer Jul 4, 2024
1e5b7db
merge 2.0
TheTechromancer Jul 4, 2024
7a1ab20
Merge branch 'bbot-2.0' into yara-excavate
liquidsec Jul 5, 2024
76b1081
missing word
liquidsec Jul 5, 2024
661331f
removing excavate on RAW_TEXT (raw now)
liquidsec Jul 5, 2024
087c65c
removing remaining parse_list_string call
liquidsec Jul 5, 2024
2c14114
black
liquidsec Jul 5, 2024
7560cc8
adding doc strings for new excavate functionality
liquidsec Jul 5, 2024
5ded168
more docstring details
liquidsec Jul 5, 2024
3e52d62
adding excavate IP url test
liquidsec Jul 5, 2024
bf71ba2
adding excavate IP url test (cleanup)
liquidsec Jul 5, 2024
41ca84f
black
liquidsec Jul 5, 2024
a528eb4
update architecture
TheTechromancer Jul 5, 2024
8c7f0cd
better error handling
TheTechromancer Jul 5, 2024
7807738
zmq .term() instead of .destroy()
TheTechromancer Jul 6, 2024
61a39e8
add gen_chord_data.py
TheTechromancer Jul 6, 2024
29f521b
fix import error
TheTechromancer Jul 6, 2024
869bf41
Merge pull request #1471 from blacklanternsecurity/yara-excavate
TheTechromancer Jul 6, 2024
1811a09
fix dns tests
TheTechromancer Jul 6, 2024
43c0f84
merge bbot 2.0
TheTechromancer Jul 6, 2024
4b854ce
fix merge conflict
TheTechromancer Jul 6, 2024
f7073b5
fix dnscaa tests
TheTechromancer Jul 6, 2024
f8c711c
Merge pull request #1530 from blacklanternsecurity/module-management-…
TheTechromancer Jul 6, 2024
3101265
reorganize web config
TheTechromancer Jul 6, 2024
0b215a4
fix tests
TheTechromancer Jul 6, 2024
dd36e95
fix cli tests
TheTechromancer Jul 6, 2024
08d0dbc
fix cli tests
TheTechromancer Jul 6, 2024
c895f34
fix command tests
TheTechromancer Jul 6, 2024
ee13c0d
fix config tests
TheTechromancer Jul 6, 2024
a80c427
fix curl tests
TheTechromancer Jul 6, 2024
5f1bfd9
fix web tests
TheTechromancer Jul 7, 2024
88add0a
fix portscan test
TheTechromancer Jul 7, 2024
d6382e3
fix unstructured tests
TheTechromancer Jul 7, 2024
24313ce
Merge pull request #1419 from blacklanternsecurity/bbot-2.0-docs
TheTechromancer Jul 7, 2024
cceecc5
add RAW_DNS_RECORD support
TheTechromancer Jul 7, 2024
f6c7bd5
add dns helpers.py
TheTechromancer Jul 7, 2024
bc4483f
fixing multuple same header extraction
liquidsec Jul 7, 2024
3fa0faa
remove debug
liquidsec Jul 7, 2024
bf8c98d
Merge pull request #1532 from blacklanternsecurity/raw-dns-records
TheTechromancer Jul 7, 2024
83f8bb4
fixing host_header module
liquidsec Jul 7, 2024
a461dcf
fixing ntlm module
liquidsec Jul 7, 2024
65cf455
fixing wappalyzer
liquidsec Jul 7, 2024
0784bcc
black
liquidsec Jul 7, 2024
52d06c5
rename dns --> dnsresolve, cloud --> cloudcheck
TheTechromancer Jul 7, 2024
03c77e5
only index dupes
liquidsec Jul 7, 2024
32703f0
bump baddns 1.1.796
liquidsec Jul 7, 2024
808584c
fixing header process function
liquidsec Jul 7, 2024
e33cbd5
fix cli tests
TheTechromancer Jul 7, 2024
284cebb
fix dns tests
TheTechromancer Jul 8, 2024
251aaa6
more work on tests
TheTechromancer Jul 8, 2024
2fe45ff
fix dnscommonsrv tests
TheTechromancer Jul 8, 2024
910cbd5
Merge pull request #1534 from blacklanternsecurity/http_response-head…
liquidsec Jul 8, 2024
aa949b0
Merge pull request #1535 from blacklanternsecurity/rename-dns-cloud
TheTechromancer Jul 8, 2024
db99e0a
use request_batch helper with bucket modules
TheTechromancer Jul 8, 2024
0289e3f
update how_it_works
TheTechromancer Jul 8, 2024
3012f5b
update how_it_works
TheTechromancer Jul 8, 2024
1fc9b9e
excavate target parameter extraction / new tests
liquidsec Jul 8, 2024
64d595b
fix firebase tests
TheTechromancer Jul 8, 2024
e0b5d11
Merge pull request #1536 from blacklanternsecurity/bucket-request-batch
TheTechromancer Jul 8, 2024
c6a0bb1
pre-release warning, better tests for raw dns records
TheTechromancer Jul 8, 2024
97445a0
fix minor http error
TheTechromancer Jul 8, 2024
803c8ba
Merge pull request #1541 from blacklanternsecurity/dns-raw-fix
TheTechromancer Jul 8, 2024
482bba8
Merge pull request #1537 from blacklanternsecurity/excavate-new-tests
liquidsec Jul 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ jobs:
publish_docs:
needs: update_docs
runs-on: ubuntu-latest
if: github.event_name == 'push' && (github.ref == 'refs/heads/dev')
if: github.event_name == 'push' && (github.ref == 'refs/heads/stable')
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
Expand Down
Loading
Loading