Any way to optimize the memory usage of Era?

[windycrypto]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

nightly-2024-08-01

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

We are running the external nodes for Era, seems the external node consumes lots of memory (>30GB), is there a way to optimize the memory usage of the external node. We have external node for cronos zkevm, zklink and the Zero chain too, these chains cost 4-8GB memory, so could we have a technical answer where does the >30GB memory consumes in Era?

Expected Behavior

the external node cost much less memory

Current Behavior

the external node cost >30GB memory

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[gleenai[bot]]

The high memory usage of the ZKsync Era external node, often exceeding 30GB, is primarily due to the complex operations and data structures required to support the zkEVM and its associated functionalities. Unlike simpler EVM-based chains, ZKsync Era's architecture involves more intensive computation and state management to ensure the security and efficiency of zero-knowledge proofs.

Here are a few technical insights into why ZKsync Era nodes may require more memory:

• State Size and Management: ZKsync Era nodes handle a larger state size due to the additional data required for zero-knowledge proofs.
• Batch Processing: Nodes may need to process and store large batches of transactions efficiently, which requires significant memory.
• Data Structures: Advanced data structures that are optimized for fast access and updates can also consume more memory.

Currently, there isn't a documented method to significantly reduce the memory usage without potentially compromising performance or security. However, you can ensure that your node is running on an optimized configuration and that unnecessary services or processes are disabled to minimize memory usage.

For more detailed technical support and potential optimizations, you might want to engage directly with the ZKsync developers community on GitHub discussions. They could provide more specific advice or updates on memory optimization efforts.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[JackHamer09]

Hey, thanks for submitting this question! I'll try to figure that out and get back to you asap.

[JackHamer09]

@windycrypto

The high memory usage (>30GB) for Era external nodes is primarily due to the Merkle tree. Specifically, a significant amount of RAM is consumed by indexes and Bloom filters for RocksDB, which are loaded into memory at node startup and pinned there by default.

To optimize memory usage, you can configure the following parameters:

• merkle_tree_block_cache_size_mb: Set this to at least 4GB.
• merkle_tree_include_indices_and_filters_in_block_cache: Set this to true.

These settings can be applied via environment variables or file-based configurations, depending on your setup. You can refer to the relevant configuration code here.

Additional optimizations:

• Pruning: This reduces the Merkle tree size and can help with memory usage. Refer to the pruning guide.
• Treeless mode: Running a node without the Merkle tree entirely is another option. However, this is experimental and not always applicable. Details are available here.

Note: These recommendations are based on profiling conducted ~6 months ago, so results may vary slightly depending on the current state of the software.

[dutterbutter]

@windycrypto I see you've recently selected unanswered here. Can you provide new details to your issue?

[windycrypto]

Hi @dutterbutter I just enabled the following two options and also enabled tree less mode

merkle_tree_block_cache_size_mb: Set this to at least 4GB.
merkle_tree_include_indices_and_filters_in_block_cache: Set this to true.

But seems it doesn't help, the node's memory usage just increase slowly(across 3-4 days) to 34GB again

[dutterbutter]

I see. Can you confirm you treeless mode is enabled? Specifically are you monitoring tree fetcher metrics? If so can you provide them?

Relevant docs: https://matter-labs.github.io/zksync-era/core/latest/guides/external-node/09_treeless_mode.html

[windycrypto]

not sure how to expose the metrics but I did follow the document and set the --components=core,api,tree_fetcher option, and I did see the tree fetcher related logs:

era-geth-1  | ESC[2m2025-02-20T07:30:24.053024ZESC[0m ESC[32m INFOESC[0m ESC[2mzksync_node_framework::service::contextESC[0mESC[2m:ESC[0m Layer tree_data_fetcher_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
era-geth-1  | ESC[2m2025-02-20T07:30:24.053025ZESC[0m ESC[32m INFOESC[0m ESC[2mzksync_node_framework::service::contextESC[0mESC[2m:ESC[0m Layer tree_data_fetcher_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
era-geth-1  | ESC[2m2025-02-20T07:30:24.053028ZESC[0m ESC[32m INFOESC[0m ESC[2mzksync_node_framework::service::contextESC[0mESC[2m:ESC[0m Layer tree_data_fetcher_layer has requested resource common/eth_interface of type zksync_node_framework::implementations::resources::eth_interface::EthInterfaceResource
era-geth-1  | ESC[2m2025-02-20T07:30:24.053031ZESC[0m ESC[32m INFOESC[0m ESC[2mzksync_node_framework::service::contextESC[0mESC[2m:ESC[0m Layer tree_data_fetcher_layer has requested resource common/gateway_eth_interface of type zksync_node_framework::implementations::resources::eth_interface::GatewayEthInterfaceResource, but it is not available
era-geth-1  | ESC[2m2025-02-20T07:30:24.053032ZESC[0m ESC[32m INFOESC[0m ESC[2mzksync_node_framework::service::contextESC[0mESC[2m:ESC[0m Layer tree_data_fetcher_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
era-geth-1  | ESC[2m2025-02-20T07:30:24.053036ZESC[0m ESC[32m INFOESC[0m ESC[2mzksync_node_framework::implementations::resources::poolsESC[0mESC[2m:ESC[0m Provided a new copy of an existing master unbound pool
era-geth-1  | ESC[2m2025-02-20T07:30:24.053041ZESC[0m ESC[33m WARNESC[0m ESC[2mzksync_node_framework::implementations::layers::tree_data_fetcherESC[0mESC[2m:ESC[0m Running tree data fetcher (allows a node to operate w/o a Merkle tree or w/o waiting the tree to catch up). This is an experimental feature; do not use unless you know what you're doing
era-geth-1  | ESC[2m2025-02-20T07:30:24.055488ZESC[0m ESC[32m INFOESC[0m ESC[2mzksync_node_framework::service::contextESC[0mESC[2m:ESC[0m Layer tree_data_fetcher_layer has added a new task: tree_data_fetcher

[dutterbutter]

Let me ask some folks internally and get back to you as it looks like it is enabled.