Misc bugfixes, asyncio speed optimizations

[TheTechromancer]

The purpose of this branch was to experiment with some asyncio-related speed optimizations in response to this issue.

The changes are designed to adapt BBOT to asyncio's idiosyncrasies by cutting down on usage of asyncio.create_task(). This reduces DNS parallelism, but speeds up performance considerably in the long run, since it frees the event loop to do other things.

Also included in this PR is the deletion of our custom CacheDict class, which was used for DNS caching and was taking up an unnecessary amount of CPU during large scans. In its place we are now using cachetools, an off-the-shelf library that's stable and much better-behaved.

Benchmark

Command:

bbot -t comcast.com -f subdomain-enum -rf passive safe

Before: 24+ hours
After: 55 minutes, 29 seconds

To help with DNS/HTTP debugging, log.trace() is now used instead of log.debug(). This lets us enable DNS/HTTP debugging independently without also needing to enable --debug.

[codecov]

Codecov Report

Attention: 63 lines in your changes are missing coverage. Please review.

Comparison is base (ac53b32) 92% compared to head (28f24ac) 92%.
Report is 47 commits behind head on dev.

Files	Patch %	Lines
bbot/cli.py	10%	38 Missing ⚠️
bbot/modules/massdns.py	77%	14 Missing ⚠️
bbot/modules/masscan.py	25%	3 Missing ⚠️
bbot/core/helpers/depsinstaller/installer.py	0%	2 Missing ⚠️
bbot/modules/internetdb.py	72%	2 Missing ⚠️
bbot/core/helpers/dns.py	93%	1 Missing ⚠️
bbot/core/helpers/misc.py	95%	1 Missing ⚠️
bbot/core/helpers/web.py	75%	1 Missing ⚠️
bbot/modules/deadly/nuclei.py	86%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #1051   +/-   ##
=====================================
- Coverage     92%     92%   -0%     
=====================================
  Files        304     306    +2     
  Lines      19285   19376   +91     
=====================================
+ Hits       17726   17778   +52     
- Misses      1559    1598   +39     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TheTechromancer]

Before we merge this we need to make sure to test with httpx/gowitness as @aconite33 reported they weren't working well on this branch.

[TheTechromancer]

Confirmed gowitness does seem to be running into some problems.