Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Misc bugfixes, asyncio speed optimizations #1051

Merged
merged 41 commits into from
Feb 14, 2024
Merged
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
d8a6418
don't create asyncio tasks for dns stuff
TheTechromancer Feb 2, 2024
ce75405
fixed resolve_batch bug
TheTechromancer Feb 2, 2024
60e095f
multiprocessize collapse_url
TheTechromancer Feb 2, 2024
b6babc7
logging for wildcards
TheTechromancer Feb 2, 2024
a060815
make sure things are awaited
TheTechromancer Feb 2, 2024
0e6254e
fix bug in dns.py
TheTechromancer Feb 2, 2024
bf539b3
flaked
TheTechromancer Feb 2, 2024
19fe685
fix rare dns bug, verbosify abort_if
TheTechromancer Feb 2, 2024
126e39b
limit anubisdb due to excessive garbage results
TheTechromancer Feb 2, 2024
99042a8
clean up code
TheTechromancer Feb 2, 2024
9c92e93
blacked
TheTechromancer Feb 2, 2024
43133a1
remove custom cache, use cachetools
TheTechromancer Feb 2, 2024
d2fbaf5
increase max dnscommonsrv handlers, small masscan bugfix
TheTechromancer Feb 2, 2024
2958a3b
massdns speed optimizations
TheTechromancer Feb 3, 2024
fd984cd
dnscommonsrv rework, spellchecking
TheTechromancer Feb 3, 2024
d32d98d
fix tests
TheTechromancer Feb 3, 2024
d11ad06
fix scope accuracy tests
TheTechromancer Feb 3, 2024
5da0e43
just telerik things
TheTechromancer Feb 4, 2024
834ed0c
increase dnscommonsrv threads
TheTechromancer Feb 4, 2024
9bbd6d4
limit massdns brute force depth
TheTechromancer Feb 4, 2024
cab606d
small wildcard tweak
TheTechromancer Feb 4, 2024
bca67d8
internetdb speed optimization
TheTechromancer Feb 4, 2024
ed36be5
massdns tweaks
TheTechromancer Feb 5, 2024
d89cda2
fix internetdb bug
TheTechromancer Feb 5, 2024
4be75af
fix \s warning
TheTechromancer Feb 5, 2024
1c5a234
increase massdns qsize
TheTechromancer Feb 5, 2024
86c0171
increase qsize for speculate and excavate
TheTechromancer Feb 6, 2024
e02c686
log version command: verbose() --> trace()
TheTechromancer Feb 6, 2024
e9cb4fd
allow independent http/dns debugging (without needing -d)
TheTechromancer Feb 6, 2024
bfef473
fix trace
TheTechromancer Feb 6, 2024
2f788d7
default qsize --> 1000, unlimited qsize for speculate & excavate
TheTechromancer Feb 7, 2024
ae25d8a
fix aioconsole bug
TheTechromancer Feb 9, 2024
c151411
more aioconsole bugfixing
TheTechromancer Feb 9, 2024
381f12a
fixed console logic
TheTechromancer Feb 9, 2024
9498117
remove unneeded cancel logic
TheTechromancer Feb 9, 2024
a1d4f0d
flaked
TheTechromancer Feb 9, 2024
7ec6e9f
debug --> verbose for batch event handling
TheTechromancer Feb 12, 2024
07eea20
keyboard listen logic fix
TheTechromancer Feb 12, 2024
29dce2f
debug massdns mutations
TheTechromancer Feb 12, 2024
ce95ce8
add debug statements for mutations
TheTechromancer Feb 13, 2024
28f24ac
restore wildcard rdtype optimization
TheTechromancer Feb 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix aioconsole bug
  • Loading branch information
TheTechromancer committed Feb 9, 2024
commit ae25d8a0d532fe31258005b77aca6a36bb289029
32 changes: 20 additions & 12 deletions bbot/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
import asyncio
import logging
import traceback
from aioconsole import ainput
from omegaconf import OmegaConf
from contextlib import suppress
from aioconsole.stream import NonFileStreamReader

# fix tee buffering
sys.stdout.reconfigure(line_buffering=True)
Expand All @@ -20,6 +20,7 @@
from bbot import __version__
from bbot.modules import module_loader
from bbot.core.configurator.args import parser
from bbot.core.helpers.misc import smart_decode
from bbot.core.helpers.logger import log_to_stderr
from bbot.core.configurator import ensure_config_files, check_cli_args, environ

Expand Down Expand Up @@ -321,21 +322,28 @@ def handle_keyboard_input(keyboard_input):
toggle_log_level(logger=log)
scanner.manager.modules_status(_log=True)

reader = NonFileStreamReader(sys.stdin)

async def akeyboard_listen():
allowed_errors = 10
while 1:
keyboard_input = "a"
try:
keyboard_input = await ainput()
except Exception:
allowed_errors -= 1
handle_keyboard_input(keyboard_input)
if allowed_errors <= 0:
break
try:
allowed_errors = 10
while 1:
keyboard_input = None
try:
keyboard_input = smart_decode((await reader.readline()).strip())
allowed_errors = 0
except Exception:
allowed_errors -= 1
if keyboard_input is not None:
handle_keyboard_input(keyboard_input)
if allowed_errors <= 0:
break
except Exception as e:
log_to_stderr(f"Error in keyboard listen task: {e}", level="ERROR")
log_to_stderr(traceback.format_exc(), level="TRACE")

try:
keyboard_listen_task = asyncio.create_task(akeyboard_listen())

await scanner.async_start_without_generator()
finally:
keyboard_listen_task.cancel()
Expand Down
Loading