memory consumption (exit code 137)

[rgaudin]

This is not my first time of seeing this ; now I am sure there are no moving pieces involved so it's time for a ticket.

this zimit run exhausted the 3.75GB or RAM during warc2zim, so after the browser process has been closed.

warc2zim should not consume this much memory of course. Binary data should be passed in chunk and the data it manipulates (to-rewrite text) should not be this large.

Given this is a scrape of one of our mirrors (🙄), either we are loading binaries in memory or some content have incorrect types leading to them being loaded for rewriting (or both). To be investigated.

[benoit74]

A "more interesting" case: https://farm.zimit.kiwix.org/pipeline/dec7c862-dafa-4f5f-b0a4-0bc2a3a81200/debug which is crawling what looks like a more legitimate website https://crustywindo.ws/