Big Block Issue Optimization

[yoozo]

Description

Background:
Currently, if a large block is encountered, the SDK may fail to index properly.
For example, Kusama: 25845986.

It behaves differently under HTTP and WS, and here’s the information we know so far:

• When using WS, it may cause an out-of-memory (OOM) error because the block is too large (at this point, we cannot do anything, and the process will be blocked), potentially leading to a 1006 connection closed error.
• When using HTTP, it may fail due to too many concurrent requests, resulting in many HTTP request failures.

Issues:

• How to detect large blocks?
• WS cannot retrieve large block results properly, causing OOM.
• HTTP cannot handle too many concurrent requests for large blocks, leading to timeouts and connection closures.

Optimizations:

• Implement an RPC response size detection mechanism. If the response exceeds the set size, record this block in the metadata, indicating that the current block height is a large block, which may warrant increasing the timeout.
• Add logs during the processing of large blocks to notify that the process is still running normally.

[stwiname]

Some more information.

Example block:
https://kusama.subscan.io/block/25845986
https://polkadot.js.org/apps/#/explorer/query/0xa5d5bddd0e6a478a54535f8988113d93d70f9e1f41371bc33449afbd0f9c15a6

Logs:

2024-11-19 11:35:57        RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: No response received from RPC endpoint in 60s
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch events at block 0x8d35c22763788ec8ad467699b75e735f856fca1a3c0dacc5b53ad34f126fa4d3
2024-11-19 11:35:57        RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch events at block 0x0f1780830c6f485c2c9dc14d6e476746fe59b8f57d830d723b8cc072e585bc99
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch events at block 0x24032d6e4ed3bd87cf10c27f3ea7073908aa42ace034382b94025837cc0e2a79
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461156
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461161
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461147
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461146
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461166
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461158
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461169
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461165

[wangminqi]

Having this issue too.
not only Kusama, Polkadot too. Around this period of blocks have ~5000 extrinsic per block.
Sample polkadot block: 23527798

[wangminqi]

I see the team is working on it. #2611

By the way, this issue is actually more urgent than it sounds. With the Spammening event of polkadot incoming, it will definitely error out all procedure if we can not handle big size issue properly.