High resource usage for docker containers

[evan-forbes]

It has been observed by one of the community members that docker containers can result in using large amounts of ram. We should investigate this.

[evan-forbes]

after further discussion with users and some experiments a large portion of this seems to be very related to the spam that was done on mocha and only for nodes that have the tx-indexer on.

This was found by comparing one node that was not experiencing the issue, and another that was in pyroscope to examine what was eating up so much ram. The node that had the tx-indexer enabled and synced from scratch (therefore including the spam in the tx-index). The node that did not experience the issue synced from scratch, but didn't enable the tx-index until after the period where the chain was spammed.

after this analysis I feel comfortable shutting down this issue. I do however think this issue emphasizes the need to switch away from something like the default tx-indexer, or alter the schema of tx storage

[rootulp]

Note: celestia-app overrides the default config so that the tx indexer is disabled. Therefore, we only expect this issue for nodes that explicitly enable the tx indexer.

Ref: https://github.com/celestiaorg/celestia-app/blame/e5e98d8f3fd5a1a8ee2b6b1751a12bb2af1eaf5f/app/default_overrides.go#L252