Filedownload Module

[TheTechromancer]

Adds a module that downloads common/interesting file types (such as PDFs and Word documents) out to a scope distance of 1. The files are downloaded into a subdirectory in the scan's output folder.

Executing the following scan will automatically spider every subdomain up to depth 2 and download any interesting file it encounters. It will even detect files without extensions by the content-type, and download them too.

bbot -t evilcorp.com -f subdomain-enum -m filedownload -c web_spider_distance=2 web_spider_depth=2

This module should serve as a good jumping-off point for a FOCA-like metadata extraction module.

During development of this module, an SSL-related error in the download() helper was discovered. The fix is included.

Note: this PR depends on #692.

[codecov]

Codecov Report

Merging #774 (70c00c5) into dev (09f492c) will increase coverage by 1%.
The diff coverage is 90%.

@@          Coverage Diff           @@
##             dev    #774    +/-   ##
======================================
+ Coverage     92%     92%    +1%     
======================================
  Files        284     286     +2     
  Lines      16863   16992   +129     
======================================
+ Hits       15382   15518   +136     
+ Misses      1481    1474     -7     

Files	Coverage Δ
bbot/modules/sslcert.py	70% <100%> (-1%)	⬇️
...st_step_2/module_tests/test_module_filedownload.py	100% <100%> (ø)
bbot/modules/filedownload.py	99% <99%> (ø)
bbot/core/helpers/web.py	92% <74%> (+1%)	⬆️

... and 2 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more