🐛 Block production sometimes takes minutes

[Thomasvdam]

🐛 Bug Report

We notice that on a few occasions the production/finalising of a block took way more than the ~7 seconds we expect.

Steps to Reproduce

Unknown, but it can be observed on Devnet. We suspect tally VM execution to play a role.

Code snippet to reproduce

N.A.

Stack trace & error message

N.A.

Expected Behavior

Block production time should be relatively consistent, especially when the load is consistent.

Your Environment

• Chain version: v0.3.0-dev.3
• Machine: t3.large

[Thomasvdam]

We suspect the timeout in the tally vm might cause/contribute to the issues we're observing.

[Thomasvdam]

With the deployed debug builds from Friday September 13th we have not observed block lag over the weekend despite a data request being issued and resolved every 10 minutes. We do see that memory usage is steadily climbing in both nodes

(the last spike is due to to block lag check script)

[Thomasvdam]

Keeping notes in our Notion: https://www.notion.so/sedaprotocol/2024-09-Block-Production-Lag-107a68d575ca801c9673d1c08824c901

[Thomasvdam]

Got the durations between all SIGABRTs for the past 10 days, rounded to make it a little easier to digest:

{
  "<1 hour": 13,
  "1 hour": 29,
  "2 hours": 24,
  "3 hours": 23,
  "4 hours": 19,
  "5 hours": 3
}