500 - Internal Server Error

[frakman1]

The web server was running fine for days, then I added a zip of some source code (curl) to test the source code static analysis plugin. I selected all plugin checkmarks.
When I came back a few hours later, the webpage show the error:

Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

The docker logs showed many errors like this:

[2021-10-30 16:16:19][docker][WARNING]: [source_code_analysis]: encountered process error while processing
[2021-10-30 16:16:20][docker][WARNING]: [source_code_analysis]: encountered process error while processing
ERROR: Could not connect to clamd on LocalSocket /var/run/clamav/clamd.ctl: No such file or directory
ERROR: Could not connect to clamd on LocalSocket /var/run/clamav/clamd.ctl: No such file or directory
ERROR: Could not connect to clamd on LocalSocket /var/run/clamav/clamd.ctl: No such file or directory
ERROR: Could not connect to clamd on LocalSocket /var/run/clamav/clamd.ctl: No such file or directory
[2021-10-30 16:16:21][docker][WARNING]: [source_code_analysis]: encountered process error while processing

for a long time leading up to a more fatal error like this:

[2021-10-30 16:16:22][Analysis][INFO]: Analysis Completed:
UID: 914684dae82a1ef21e8343a7a5edd90850916439c1a250b9d164bd16258ec37e_3250
 Processed analysis: ['unpacker', 'file_type', 'crypto_hints', 'binwalk', 'users_and_passwords', 'printable_strings', 'malware_scanner', 'software_components', 'ip_and_uri_finder', 'crypto_material', 'qemu_exec', 'file_system_metadata', 'input_vectors', 'interesting_uris', 'string_evaluator', 'source_code_analysis', 'exploit_mitigations', 'cpu_architecture', 'elf_analysis', 'file_hashes', 'cve_lookup', 'init_systems', 'kernel_config', 'hardware_analysis', 'cwe_checker', 'known_vulnerabilities', 'tlsh']
 Files included: set()
[2021-10-30 16:16:41][db_interface_backend][ERROR]: Update of analysis failed badly (connection closed)
Process ExceptionSafeProcess-54:
[2021-10-30 16:16:41][db_interface_statistic][ERROR]: Could not store statistic backend (localhost:27018: [Errno 104] Connection reset by peer)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 710, in command
    return command(self, dbname, spec, secondary_ok,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 152, in command
    reply = receive_message(sock_info, request_id)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 200, in receive_message
    _receive_data_on_socket(sock_info, 16, deadline))
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 291, in _receive_data_on_socket
    chunk_length = sock_info.sock.recv_into(mv[bytes_read:])
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/FACT_core/src/storage/db_interface_statistic.py", line 37, in update_statistic
    self.statistic.insert_one(content_dict)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 705, in insert_one
    self._insert(document,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 620, in _insert
    return self._insert_one(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 609, in _insert_one
    self.__database.client._retryable_write(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1552, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1438, in _retry_with_session
    return self._retry_internal(retryable, func, session, bulk)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1470, in _retry_internal
    return func(session, sock_info, retryable)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 597, in _insert_command
    result = sock_info.command(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 726, in command
    self._raise_connection_failure(error)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 929, in _raise_connection_failure
    _raise_connection_failure(self.address, error)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 247, in _raise_connection_failure
    raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: localhost:27018: [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/FACT_core/src/helperFunctions/process.py", line 57, in run
    raise exception
  File "/opt/FACT_core/src/helperFunctions/process.py", line 52, in run
    Process.run(self)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/FACT_core/src/scheduler/Analysis.py", line 391, in result_collector
    self.post_analysis(fw)
  File "/opt/FACT_core/src/storage/db_interface_backend.py", line 154, in add_analysis
    self._update_analysis(file_object, analysis_system, processed_analysis[analysis_system])
  File "/opt/FACT_core/src/storage/db_interface_backend.py", line 170, in _update_analysis
    raise exception
  File "/opt/FACT_core/src/storage/db_interface_backend.py", line 162, in _update_analysis
    collection.update_one(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 1028, in update_one
    self._update_retryable(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 877, in _update_retryable
    return self.__database.client._retryable_write(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1552, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1438, in _retry_with_session
    return self._retry_internal(retryable, func, session, bulk)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1470, in _retry_internal
    return func(session, sock_info, retryable)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 869, in _update
    return self._update(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 838, in _update
    result = sock_info.command(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 726, in command
    self._raise_connection_failure(error)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 710, in command
    return command(self, dbname, spec, secondary_ok,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 152, in command
    reply = receive_message(sock_info, request_id)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 200, in receive_message
    _receive_data_on_socket(sock_info, 16, deadline))
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 297, in _receive_data_on_socket
    raise AutoReconnect("connection closed")
pymongo.errors.AutoReconnect: connection closed
[2021-10-30 16:16:42][process][ERROR]: Exception in Unpacking process:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 710, in command
    return command(self, dbname, spec, secondary_ok,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 152, in command
    reply = receive_message(sock_info, request_id)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 200, in receive_message
    _receive_data_on_socket(sock_info, 16, deadline))
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 291, in _receive_data_on_socket
    chunk_length = sock_info.sock.recv_into(mv[bytes_read:])
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/FACT_core/src/helperFunctions/process.py", line 52, in run
    Process.run(self)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/FACT_core/src/scheduler/Unpacking.py", line 69, in unpack_worker
    extracted_objects = unpacker.unpack(fo)
  File "/opt/FACT_core/src/unpacker/unpack.py", line 42, in unpack
    extracted_file_objects = self.generate_and_store_file_objects(extracted_files, Path(tmp_dir.name) / 'files', current_fo)
  File "/opt/FACT_core/src/unpacker/unpack.py", line 83, in generate_and_store_file_objects
    self.db_interface.set_unpacking_lock(current_file.uid)
  File "/opt/FACT_core/src/storage/db_interface_common.py", line 314, in set_unpacking_lock
    self.locks.insert_one({'uid': uid})
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 705, in insert_one
    self._insert(document,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 620, in _insert
    return self._insert_one(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 609, in _insert_one
    self.__database.client._retryable_write(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1552, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1438, in _retry_with_session
    return self._retry_internal(retryable, func, session, bulk)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1470, in _retry_internal
    return func(session, sock_info, retryable)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 597, in _insert_command
    result = sock_info.command(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 726, in command
    self._raise_connection_failure(error)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 929, in _raise_connection_failure
    _raise_connection_failure(self.address, error)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 247, in _raise_connection_failure
    raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: localhost:27018: [Errno 104] Connection reset by peer

[2021-10-30 16:16:42][db_interface_statistic][ERROR]: Could not store statistic database (connection closed)
Traceback (most recent call last):

The error seems to be related to mongodb. It looks like it stopped working at some point because I don't see port 27018 in the netstat output anymore.

I also see messages like this in the logs

[2021-10-31 00:47:06][docker][WARNING]: [source_code_analysis]: encountered process error while processing
[2021-10-31 00:47:06][docker][WARNING]: [source_code_analysis]: encountered process error while processing

[jstucke]

pymongo.errors.AutoReconnect: localhost:27018: [Errno 104] Connection reset by peer

That error suggests that your database stopped at some point. You will probably need to restart your container (you could also try to exec into the container and only start the DB but I'm not sure if that will work).
The worker processes should generally restart in case of errors. In rare cases this could fail, though (here too restarting FACT helps).

ERROR: Could not connect to clamd on LocalSocket /var/run/clamav/clamd.ctl: No such file or directory

There seem to be some problems with FACT docker in combination with the OMS plugin / ClamAV. Did it work for you at some point?

[2021-10-30 16:16:20][docker][WARNING]: [source_code_analysis]: encountered process error while processing

This could mean that call to the "linguist" docker container did not work. Did it produce any results at all? Could be another mounting path problem.

[frakman1]

I have not used ClamAV or linguist before and not sure which category they fall in. They may never have worked and I didn't notice.

I am guessing ClamAV relates to the "malware scanner" tab and linguist is for the "source code analysis" tab?

[jstucke]

ClamAV is an open source Linux anti-malware scanner and should be used to scan the file when you run the "malware scanner" plugin

linguist is a tool that is used internally for the "source code analysis" plugin for determining the (programming) language

[jstucke]

What I meant was: were there any results for the respective plugins?

[frakman1]

• Source Code Analysis

Since I have always only run firmware analysis (not source code) I have never looked at the source code analysis tab. I just checked an older firmware analysis and opened a script file /etc/init.d/firewall and the source code analysis tab shows an error. The same as the one I opened a ticket for here
Example:

• Malware Scanner

Anytime that I have looked at that, the page returns clean results. Example: