Badsecrets erroring upon a specific URL

[ausmaster]

Describe the bug
Badsecrets dies upon hitting a certain URL.

Expected behavior
Process the event successfully.

BBOT Command
<Redacted>

OS, BBOT Installation Method + Version
<Redacted>

BBOT Config
<Redacted>

Logs
[DBUG] badsecrets.finished: False
[DBUG] running: True
[DBUG] tasks:
[DBUG] - badsecrets.handle_event(HTTP_RESPONSE("{'url': '<Redacted>', 'timestamp': '2...", module=httpx, tags={'in-scope', '<Redacted>', 'dir', 'ip-<Redacted>, 'status-200'})) running for 2 minutes, 6 seconds:
[DBUG] incoming_queue_size: 19969
[DBUG] outgoing_queue_size: 0

Screenshots
N/A

[ausmaster]

@liquidsec @TheTechromancer

[liquidsec]

Not understanding the issue, all I see is it took ~2 minutes on one particular site?

[TheTechromancer]

The issue is the ~20K events in the queue, after all other modules have finished, which suggests slowness on more than just one site. Is that fair @ausmaster ?

[ausmaster]

The issue is the ~20K events in the queue, after all other modules have finished, which suggests slowness on more than just one site. Is that fair @ausmaster ?

That actually might part of it or the issue as well, me and @liquidsec were wondering about the resources required to run this particular scan.

[TheTechromancer]

Yeah as we talked about before, BBOT is a CPU-intensive tool and depends heavily on single-threaded performance. So it's really important these VMs get good CPUs.

[TheTechromancer]

Still, I'd be curious to see the specific cause behind why certain sites are taking longer. E.g. how much time is spent on regex vs brute-forcing?

[TheTechromancer]

#2074

[liquidsec]

@TheTechromancer you ok with closing at least for now, since they were on an old version of badsecrets that was before some significant performance enhancements?

[TheTechromancer]

That's fine, we can reopen if needed

[liquidsec]

closing for now since slowdown was at least partially explained