HTTP Engine - Offload Web Requests to Dedicated Process

[TheTechromancer]

TODO:

• Code comments for:
  • request_batch
  • request_custom_batch
• Tests for async cancellation (during request_batch())
• Tests for raise_error=True
• Fix strange race condition causing duplicate IP_ADDRESS (from A module)
• Implement new request_batch() feature:
  • ntlm
  • ajaxpro
  • telerik
  • http_compare
  • paramminer
  • host_header
  • bypass403
  • iis_shortnames
• Fix ctrl+c bug

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 91.33858% with 77 lines in your changes are missing coverage. Please review.

Project coverage is 92%. Comparing base (b16f7d9) to head (bfd62de).
Report is 37 commits behind head on bbot-2.0.

❗ Current head bfd62de differs from pull request most recent head 41b8cdc. Consider uploading reports for the commit 41b8cdc to get more accurate results

Files	Patch %	Lines
bbot/core/helpers/web/engine.py	78%	41 Missing ⚠️
bbot/core/engine.py	89%	14 Missing ⚠️
bbot/core/helpers/web/web.py	93%	5 Missing ⚠️
bbot/core/helpers/dns/engine.py	43%	4 Missing ⚠️
bbot/modules/iis_shortnames.py	92%	4 Missing ⚠️
bbot/scanner/target.py	96%	3 Missing ⚠️
bbot/core/core.py	75%	2 Missing ⚠️
bbot/modules/ntlm.py	93%	1 Missing ⚠️
bbot/modules/output/http.py	50%	1 Missing ⚠️
bbot/modules/output/splunk.py	50%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff            @@
##           bbot-2.0   #1340    +/-   ##
=========================================
+ Coverage        92%     92%    +1%     
=========================================
  Files           334     339     +5     
  Lines         22451   22901   +450     
=========================================
+ Hits          20545   21031   +486     
+ Misses         1906    1870    -36     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.