File Download Module

[TheTechromancer]

Adds a module that downloads common/interesting file types (such as PDFs and Word documents) out to a scope distance of 1. The files are downloaded into a subdirectory in the scan's output folder.

Executing the following scan will automatically spider every subdomain up to depth 2 and download any interesting file it encounters. It will even detect files without extensions by the content-type, and download them too.

bbot -t evilcorp.com -f subdomain-enum -m filedownload -c web_spider_distance=2 web_spider_depth=2

This module should serve as a good jumping-off point for a FOCA-like metadata extraction module.

During development of this module, an SSL-related error in the download() helper was discovered. The fix is included.

Note: this PR depends on #692.

[codecov]

Codecov Report

Merging #754 (1b6d8c9) into dev (6e17307) will increase coverage by 1%.
Report is 72 commits behind head on dev.
The diff coverage is 91%.

❗ Current head 1b6d8c9 differs from pull request most recent head 032d3ff. Consider uploading reports for the commit 032d3ff to get more accurate results

@@          Coverage Diff           @@
##             dev    #754    +/-   ##
======================================
+ Coverage     92%     92%    +1%     
======================================
  Files        284     287     +3     
  Lines      16692   16885   +193     
======================================
+ Hits       15243   15439   +196     
+ Misses      1449    1446     -3     

Files	Coverage Δ
bbot/cli.py	88% <100%> (ø)
bbot/core/event/base.py	95% <ø> (ø)
bbot/core/event/helpers.py	90% <ø> (ø)
bbot/core/helpers/command.py	97% <ø> (ø)
bbot/core/helpers/dns.py	86% <100%> (+2%)	⬆️
bbot/core/helpers/files.py	65% <ø> (ø)
bbot/core/helpers/helper.py	98% <ø> (ø)
bbot/core/helpers/interactsh.py	78% <100%> (ø)
bbot/core/helpers/misc.py	91% <100%> (+1%)	⬆️
bbot/core/helpers/modules.py	97% <100%> (+1%)	⬆️
... and 61 more

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[aconite33]

What is the purpose of changing agent.py? Is this for downloading specifically or is it a change required in order to allow for downloading?

[TheTechromancer]

Hmm I think I must have been trying to get the tests to pass. We can test it during the troubleshooting call with @batgoose tomorrow.

There were no changes to agent.py as part of this module. The reason you're seeing here it is because we need to merge this one first.