ffuf module blocking event loop

[TheTechromancer]

The ffuf module is blocking the event loop for 3+ seconds:

$ python log_timedeltas.py /tmp/debug2.log 
Top 10 longest delays:

1. Delay: 3.658000 seconds
   From: 2024-08-18 20:57:36.972000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 Skipping adding [images] to wordlist because it was in the blacklist
   To:   2024-08-18 20:57:40.630000
   [VERBOSE] bbot.modules.ffuf_shortnames logger.py:132 generated temp word list of size [40] for filename hint: [system]

2. Delay: 3.540000 seconds
   From: 2024-08-18 21:03:02.432000
   [VERBOSE] bbot.modules.ffuf_shortnames logger.py:132 Detected delimiter [_] in hint [web_20]
   To:   2024-08-18 21:03:05.972000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 Skipping adding [css] to wordlist because it was in the blacklist

3. Delay: 3.413000 seconds
   From: 2024-08-18 21:03:34.368000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 Skipping adding [images] to wordlist because it was in the blacklist
   To:   2024-08-18 21:03:37.781000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 in mode [normal]

4. Delay: 3.411000 seconds
   From: 2024-08-18 20:52:57.947000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 Skipping adding [images] to wordlist because it was in the blacklist
   To:   2024-08-18 20:53:01.358000
   [VERBOSE] bbot.modules.ffuf_shortnames logger.py:132 generated temp word list of size [1] for filename hint: [app_gl]

5. Delay: 3.411000 seconds
   From: 2024-08-18 21:07:08.050000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 Skipping adding [images] to wordlist because it was in the blacklist
   To:   2024-08-18 21:07:11.461000
   [VERBOSE] bbot.modules.ffuf_shortnames logger.py:132 generated temp word list of size [8] for filename hint: [aspnet]

6. Delay: 3.393000 seconds
   From: 2024-08-17 03:26:18.026000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 Skipping adding [images] to wordlist because it was in the blacklist
   To:   2024-08-17 03:26:21.419000
   [VERBOSE] bbot.modules.ffuf_shortnames logger.py:132 generated temp word list of size [0] for filename hint: [2_0_50]

7. Delay: 3.392000 seconds
   From: 2024-08-18 20:56:59.881000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 Skipping adding [images] to wordlist because it was in the blacklist
   To:   2024-08-18 20:57:03.273000
   [VERBOSE] bbot.modules.ffuf_shortnames logger.py:132 generated temp word list of size [4] for filename hint: [compat]

8. Delay: 3.384000 seconds
   From: 2024-08-18 20:54:12.159000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 Skipping adding [images] to wordlist because it was in the blacklist
   To:   2024-08-18 20:54:15.543000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 in mode [normal]

9. Delay: 3.372000 seconds
   From: 2024-08-18 20:54:20.376000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 Skipping adding [images] to wordlist because it was in the blacklist
   To:   2024-08-18 20:54:23.748000
   [VERBOSE] bbot.modules.ffuf_shortnames logger.py:132 generated temp word list of size [1] for filename hint: [web_20]

10. Delay: 3.369000 seconds
   From: 2024-08-18 21:03:26.611000
   [DEBUG] bbot.modules.ffuf_shortnames base.py:1235 Skipping adding [images] to wordlist because it was in the blacklist
   To:   2024-08-18 21:03:29.980000
   [VERBOSE] bbot.modules.ffuf_shortnames logger.py:132 generated temp word list of size [1] for filename hint: [web_20]

The offending function is generate_templist(). We need to figure out why this function is so CPU-heavy and make it more efficient.

[liquidsec]

It's going through the entire word list and finding values that match a given prefix, etc. Honestly If you made me guess, I would have said it would have taken a lot longer. 3 seconds doesn't seem that bad for this. That being said, I think I can see a way to squeeze a tad more optimization out of it...

[TheTechromancer]

3 seconds is an extremely long time to block the event loop. With thousands of open TCP and UDP connections that need attention, blocking for even more than .1 second is unacceptable. Anything that blocks for longer than that needs to be executed using the process pool.

[liquidsec]

"Anything that blocks for longer than that needs to be executed using the process pool."

While I do think i could optimize it some, even with 50% better optimization that would still be over a second. So it sounds like that's what needs to happen.

[liquidsec]

Well, I did even a little better than 50% i think...

Against my .NET vulnerability playground designed to test ffuf_shortnames...

before #1696 :

after:

if you still want to put it in process jail feel free but this should help a lot I think.

[liquidsec]

should be at least partially addressed with #1696

[TheTechromancer]

Tested ffuf_shortnames against a live target and got these results for generate_templist():

avg, before optimization (stable): .032 seconds 2.8 seconds
avg, after optimization: 0.031 seconds

@liquidsec do you know what case would have caused those high processing times of 3+ seconds?

EDIT: nevermind I'm a dummy.