Dastardly scanner

[domwhewell-sage]

As mentioned in #895 created a dastardly scanner module.

Dastardly is a lightweight web application security scanner, Originally designed for CI/CD pipelines but added as a module to bbot to allow for the detection of Reflected XSS, CORS, Vulnerable JS, Content type not specified, Multiple Content Types, HTML Charset and duplicate cookies. 7 common web application security issues.

The module is tagged as aggressive and the --deadly flag should be used to activate it.

It will take in_scope_only and URL events as input and start the dastardly docker container to crawl and audit the url. Dastardly crawling is limited to 10 minutes per url. But with many URLs and sub domains being discovered by other modules this should give a decent coverage of any web application.

Currently the description of each finding/vulnerability would be "Cross-site scripting (reflected) found at https://ginandjuice.shop/catalog/subscribe" (Replace "Cross-site scripting (reflected)" with the finding name and "https://ginandjuice.shop/catalog/subscribe" with the exact URL) as the evidence provided by dastardly is too long to fit into the description which is included in the output.csv.

• Not yet implemented is a way of displaying the evidence in the correct format but it may be possible to create individual evidence files for each dastardly finding

A test case has been added also to run the dastardly module against portswiggers deliberately vulnerable web app https://ginandjuice.shop/. (The test case also enables the module nmap to provide TCP_OPEN events and httpx to provide URL events)

[TheTechromancer]

Thanks; this looks like a well-crafted PR. I will start reviewing today.

[codecov]

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (15d0798) 92% compared to head (6007edd) 92%.

Files	Patch %	Lines
bbot/modules/deadly/dastardly.py	96%	3 Missing ⚠️
.../test_step_2/module_tests/test_module_dastardly.py	95%	2 Missing ⚠️
bbot/core/helpers/depsinstaller/installer.py	67%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##             dev    #896    +/-   ##
======================================
+ Coverage     92%     92%    +1%     
======================================
  Files        290     292     +2     
  Lines      18018   18134   +116     
======================================
+ Hits       16559   16673   +114     
- Misses      1459    1461     +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TheTechromancer]

@domwhewell-sage excellent work on this PR.

I made a few small updates:

• Rely on local web server for tests (for better speed/reliability during tests)
• Docker dependency accounts for differences in common linux distros (redhat, debian, arch)
• Renamed dastardly's .xml output to include the host, do not delete at end of scan (so the user can go back and review if needed)
• Consume HTTP_RESPONSE instead of URL. (In practice this is the same, except it allows us to more easily filter on response code)
• Reject 302 response codes to avoid scanning the same host twice (e.g. HTTP redirecting to HTTPS)
• Add tags "slow", "web-thorough"
• Perform only one scan per host+port

Please let me know if you're opposed to any of these and feel free to make any updates. Then we can pass it off to @liquidsec for review.

[domwhewell-sage]

That all looks good to me @TheTechromancer

[TheTechromancer]

Working on getting tests to pass. Running docker inside github action may be problematic.

[domwhewell-sage]

Hmm deps_apt and deps_shell were originally working I'm sure tests passed on 39 minutes on the first run.

The only main difference now is systemctl I think. I'm sure the main dastardly GitHub action uses their docker file to run

[TheTechromancer]

Ahh you're right. Looks to be working now; that's a relief.