ENOMEM In very specific cases

[svidovich]

In certain use cases, I will get less files than expected from a particular firmware. According tot he container logs, the unpacker process ( whichever it might be -- binwalk, for example ) has run the machine out of memory and been killed before it completed. Because the logic for successful unpack is whether or not there are files resultant, if binwalk runs and doesn't finish but does manage to produce files, the unpack will have been deemed 'successful' by the plugin, but in reality it was not successful.

Now, it is noted that the plugins are calling execute_shell_command from common_helper_process. execute_shell_command seems to be dropping exit codes on the floor. Could it be possible to instead use execute_shell_command_get_return_code instead, and handle the case in which a process returns something non-zero?

By the way the plugins return data, it seems to be a dictionary, so adding the return code into the dictionary along with the output shouldn't break anything. Perhaps something can be done in unpack.py to handle this? I am willing to help in this effort.

[dorpvom]

Hi, sounds useful to me. I have not thought about memory issues with docker yet so an indicator for something out of the ordinary would definitely help.
If you like to contribute I would be happy to review a Pull Request.

[dorpvom]

In the meantime, can you make out a trend for when this happens? I would guess on large / very large files but there could be a different problem altogether.

[svidovich]

Yes, it is with large files, especially those with large compression ratios. Binwalk has been the child process so far that I've witnessed issues with. I imagine it doesn't help that the container has been constrained pretty tightly -- it has only been given 512MB of memory to work with!

[svidovich]

I made a pull request in regards that should mitigate this problem somewhat...

[dorpvom]

So, first thanks for the contribution. If you can incorporate the requests in the PR the changes will be merged.
Regarding the generally low memory for containers: This can be chose when running the container, i.e. with the docker run command. So from my viewpoint this is a FACT, not FACT_extractor problem. I will add an option to the interface in #10 and consider a config value for the FACT PR.