CPU Usage Inquisition

[TheTechromancer]

CPU resources in the main BBOT process are extremely scarce, which clogs the event loop during larger scans. We need to prioritize finding these big offenders which are taking up a lot of CPU, and migrate them out of the main BBOT process.

Notice how in this cProfile benchmark, the very top offender is method 'search' of 're.Pattern' objects. This should not be anywhere near the top! Any CPU-intensive regexes should NOT be using the re library, but using our asyncified regex helpers instead.

   Ordered by: internal time                                                                                                                                                                                                                                                   
   List reduced from 11737 to 50 due to restriction <50>                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                               
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)                                                                                                                                                                                                        
  3999386  713.252    0.000  713.252    0.000 {method 'search' of 're.Pattern' objects}                                                                                                                                                                                        
  6151312  472.257    0.000  600.272    0.000 {method 'match' of '_regex.Pattern' objects}                                                                                                                                                                                     
  4541684  166.778    0.000  203.664    0.000 /root/bbot/bbot/core/event/base.py:948(__str__)                                          
1276892/1274389  160.049    0.000  339.462    0.000 {method 'search' of '_regex.Pattern' objects}                                                                                                                                                                              
6202814/6202776  140.969    0.000  187.950    0.000 {built-in method posix.urandom}                                                    
2395220/2304333   98.534    0.000  272.969    0.000 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/Wappalyzer/Wappalyzer.py:240(_has_technology)                                                                                          
   158118   77.279    0.000  289.497    0.002 /root/bbot/bbot/core/helpers/misc.py:2489(weighted_shuffle)                                                                                                                                                                      
1991942/1956238   76.198    0.000   81.313    0.000 {method 'sub' of '_regex.Pattern' objects}                                         
12157714/12157698   67.504    0.000  130.222    0.000 {built-in method builtins.sum}                                                   
229239261/229227000   61.379    0.000   88.000    0.000 {built-in method builtins.isinstance}                                          
450478182   59.695    0.000   60.285    0.000 /root/bbot/bbot/core/helpers/misc.py:2523(<genexpr>)                                                                                                                                                                             
25461/25250   58.747    0.002 4619.316    0.183 {method 'poll' of 'select.epoll' objects}                                              
 11700741   54.614    0.000   71.195    0.000 /usr/lib/python3.12/random.py:454(choices)                                               
  76296/9   47.198    0.001 4557.623  506.403 {method 'get' of '_queue.SimpleQueue' objects}                                           
 18394814   38.803    0.000  107.413    0.000 /usr/lib/python3.12/ipaddress.py:1187(_ip_int_from_string)                                                                                                                                                                       
 14200921   38.746    0.000   58.127    0.000 /usr/lib/python3.12/ipaddress.py:1651(_ip_int_from_string)                                                                                                                                                                       
 10367322   38.547    0.000  189.311    0.000 /usr/lib/python3.12/ipaddress.py:28(ip_address)                                          
 18527341   34.788    0.000  154.705    0.000 /usr/lib/python3.12/ipaddress.py:1286(__init__)                                          
 14271211   34.167    0.000  115.714    0.000 /usr/lib/python3.12/ipaddress.py:1920(__init__)                                          
168670785/168404923   32.571    0.000   32.695    0.000 {built-in method builtins.len}                                                 
  6155448   31.607    0.000   40.755    0.000 /usr/lib/python3.12/uuid.py:139(__init__)                                                
  3736741   30.496    0.000  108.122    0.000 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/idna/core.py:245(check_label)                                                                                                                
 19797282   30.361    0.000   43.332    0.000 /usr/lib/python3.12/ipaddress.py:1213(_parse_octet)                                                                                                                                                                              
   105833   27.872    0.000   52.355    0.000 {method 'findall' of '_regex.Pattern' objects}                                           
   100312   25.164    0.000   31.860    0.000 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/zmq/sugar/socket.py:626(send)                                                                                                                
90759/60650   23.716    0.000   52.493    0.001 /usr/lib/python3.12/concurrent/futures/thread.py:53(run)                                                                                                                                                                       
 17575450   23.323    0.000   45.203    0.000 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/idna/intranges.py:42(intranges_contain)                                                                                                      
  7117289   23.220    0.000  237.792    0.000 /usr/lib/python3.12/ipaddress.py:57(ip_network)                                          
  5529136   22.546    0.000  358.570    0.000 /root/bbot/bbot/core/helpers/async_helpers.py:74(__aenter__)                                                                                                                                                                     
78441/78429   21.432    0.000   25.184    0.000 {method 'flush' of '_io.TextIOWrapper' objects}                                                                                                                                                                                
 48384012   21.309    0.000   21.309    0.000 {method 'split' of 'str' objects}                                                        
  7146841   20.282    0.000  141.171    0.000 /usr/lib/python3.12/ipaddress.py:1502(__init__)                                          
   100312   20.045    0.000   21.101    0.000 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/zmq/sugar/socket.py:236(close)                                                                                                               
   100312   19.662    0.000   20.276    0.000 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/zmq/sugar/socket.py:317(connect)                                                                                                             
  3821700   19.544    0.000  250.732    0.000 /root/bbot/bbot/modules/base.py:845(queue_event)                                         
  3573219   19.110    0.000   54.291    0.000 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/omegaconf/_utils.py:746(_get_value)                                                                                                          
  7257242   18.221    0.000   45.844    0.000 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/omegaconf/_utils.py:508(_is_missing_value)                                                                                                   
   197309   17.660    0.000 3707.817    0.019 /root/bbot/bbot/modules/base.py:1579(_worker)                                            
108162/105239   17.551    0.000   20.854    0.000 {method 'send' of '_socket.socket' objects}                                          
 11573223   17.444    0.000   79.240    0.000 /usr/lib/python3.12/ipaddress.py:533(_split_addr_prefix)                                                                                                                                                                         
  9006019   16.812    0.000   16.812    0.000 {method 'replace' of 'str' objects}                                                      
  2379966   15.587    0.000  120.894    0.000 /root/bbot/bbot/core/helpers/misc.py:658(make_ip_type)                                                                                                                                                                           
   391682   15.550    0.000   37.030    0.000 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/zmq/_future.py:594(_handle_recv)                                                                                                             
   895027   15.099    0.000  304.644    0.000 /root/bbot/bbot/modules/base.py:777(_event_postcheck_inner)                                                                                                                                                                      
  5327019   15.011    0.000   72.537    0.000 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/omegaconf/dictconfig.py:455(_get_node)                                                                                                       
394979/394974   14.452    0.000   45.672    0.000 {built-in method _pickle.loads}                                                      
29782645/29782639   14.249    0.000   14.280    0.000 {built-in method _abc._abc_instancecheck}                                                                                                                                                                                
 11961098   14.066    0.000   57.966    0.000 {built-in method from_bytes}                                                             
  6155440   13.962    0.000  299.920    0.000 /usr/lib/python3.12/uuid.py:723(uuid4)                                                   
   626304   13.679    0.000  699.553    0.001 /root/bbot/bbot/core/event/base.py:124(__init__)

[TheTechromancer]

Top functions by average call time:

Average time per call (top 50)
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2    0.072    0.036    0.417    0.209 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/wordninja.py:29(__init__)
        1    0.008    0.008    0.008    0.008 ~:0(<method 'load_verify_locations' of '_ssl._SSLContext' objects>)
        1    0.007    0.007    0.007    0.007 ~:0(<method 'set_default_verify_paths' of '_ssl._SSLContext' objects>)
        1    0.007    0.007    0.008    0.008 /root/bbot/bbot/modules/digitorus.py:22(parse_results)
        1    0.005    0.005    1.846    1.846 /root/bbot/bbot/modules/secretsdb.py:28(setup)
        4    0.019    0.005    0.019    0.005 ~:0(<built-in method posix.sendfile>)
        1    0.004    0.004    0.032    0.032 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/bs4/dammit.py:70(_populate_class_variables)
        1    0.004    0.004    0.011    0.011 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/regex/_regex_core.py:1(<module>)
        1    0.004    0.004    0.004    0.004 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/pycparser/yacctab.py:1(<module>)
     1886    7.407    0.004    8.805    0.005 /root/bbot/bbot/modules/secretsdb.py:68(search_data)
        4    0.015    0.004    0.251    0.063 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/tabulate.py:1257(_wrap_text_to_colwidths)
        9    0.027    0.003    0.028    0.003 ~:0(<built-in method pydantic_core._pydantic_core.validate_core_schema>)
       14    0.040    0.003    6.100    0.436 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/cloudcheck/providers/base.py:125(update_ranges)
        1    0.003    0.003    7.833    7.833 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/cloudcheck/providers/__init__.py:45(load_from_json)
    25461   58.747    0.002 4619.316    0.181 ~:0(<method 'poll' of 'select.epoll' objects>)
        2    0.004    0.002    0.037    0.019 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/tldextract/tldextract.py:391(create)
      958    1.816    0.002   20.193    0.021 /usr/lib/python3.12/html/parser.py:134(goahead)
     3420    6.467    0.002    7.744    0.002 ~:0(<method 'match' of 'yara.Rules' objects>)
      117    0.219    0.002    0.508    0.004 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/aiosqlite/core.py:148(__await__)
        1    0.002    0.002    0.353    0.353 /root/bbot/bbot/core/modules.py:111(preload)
      656    1.141    0.002    1.149    0.002 ~:0(<built-in method posix.listdir>)
      403    0.683    0.002    0.918    0.002 ~:0(<built-in method _posixsubprocess.fork_exec>)
     8297   13.279    0.002   90.580    0.011 ~:0(<method 'poll' of 'select.poll' objects>)
        1    0.002    0.002    0.140    0.140 /root/bbot/bbot/modules/hackertarget.py:21(parse_results)
     1886    2.857    0.002    4.104    0.002 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/bs4/builder/_lxml.py:376(feed)
       70    0.105    0.001    2.796    0.040 /root/bbot/bbot/core/helpers/misc.py:1592(latest_mtime)
       68    0.100    0.001    0.134    0.002 ~:0(<built-in method posix.mkfifo>)
        1    0.001    0.001    0.003    0.003 /root/bbot/bbot/modules/crt.py:26(parse_results)
      241    0.321    0.001    2.298    0.010 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/psutil/_pslinux.py:418(virtual_memory)
        1    0.001    0.001    0.038    0.038 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/tldextract/tldextract.py:445(__init__)
       60    0.075    0.001    0.075    0.001 ~:0(<method 'feed' of 'xml.etree.ElementTree.XMLParser' objects>)
       10    0.012    0.001    0.012    0.001 ~:0(<method 'decompress' of 'zlib._ZlibDecompressor' objects>)
        1    0.001    0.001    0.005    0.005 /root/bbot/bbot/core/helpers/depsinstaller/installer.py:61(install)
       38    0.042    0.001    0.059    0.002 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/puremagic/main.py:142(_identify_all)
        4    0.004    0.001    0.388    0.097 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/tabulate.py:1280(tabulate)
        1    0.001    0.001    0.005    0.005 /root/bbot/bbot/core/helpers/wordcloud.py:327(save)
        1    0.001    0.001    0.001    0.001 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/pydantic/_internal/_known_annotated_metadata.py:1(<module>)
        3    0.003    0.001    1.065    0.355 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/baddns/lib/loader.py:12(load_signatures)
       24    0.024    0.001    0.102    0.004 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/pycparser/ply/yacc.py:1001(parseopt_notrack)
      416    0.411    0.001    0.423    0.001 ~:0(<built-in method posix.waitpid>)
        2    0.002    0.001    0.006    0.003 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/antlr4/atn/ATNDeserializer.py:200(readEdges)
       13    0.012    0.001    0.823    0.063 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/cloudcheck/providers/base.py:49(validate_cidrs)
      117    0.109    0.001    0.113    0.001 ~:0(<built-in method _sqlite3.connect>)
        1    0.001    0.001    0.214    0.214 /root/bbot/bbot/core/helpers/wordcloud.py:484(__init__)
        1    0.001    0.001    0.001    0.001 ~:0(<built-in method _pickle.load>)
        1    0.001    0.001    0.002    0.002 /root/bbot/bbot/core/helpers/wordcloud.py:303(json)
       88    0.059    0.001    0.128    0.001 /root/bbot/bbot/core/helpers/misc.py:842(extract_params_json)
     1886    1.240    0.001   15.880    0.008 /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/lib/python3.12/site-packages/Wappalyzer/Wappalyzer.py:399(analyze)
     5737    3.600    0.001    4.668    0.001 ~:0(<method 'do_handshake' of '_ssl._SSLSocket' objects>)
    76296   47.198    0.001 4557.623    0.060 ~:0(<method 'get' of '_queue.SimpleQueue' objects>)