Investigate time to consensus for really large block sizes

[evan-forbes]

Summary

As touched on in the experimental section of #381, we should further explore the consequences of creating large blocks during consensus.

Details

With our recent switch to propagating blocks using the standard tendermint block gossiping mechanism #423, we should perform some experiments to give us a better idea of the limitations of increasing the amount of block data included in each block proposed during consensus.

Action Items

• create experiment to spin up a devnet using the dummyapp
• observe the affect of increasing the amount of block data on time to consensus
• (bonus) observe the affect of altering the size of the messages gossiped using the PartSetHeader

References

tracking issue #384
more discussion on potentially different mechanisms of block propagation during consensus #434 #423
introduction of the dummyapp #384

blocked by #436

[liamsi]

Osmosis blocks are often about 2mb and with 7sec block time. During Game of Stake we experimented with larger blocks as well (according to Zaki up 20 mb with no issues). IMO, we should still test this but 4mb blocks (meaning 4mb ODS) are really sufficient. Particularly the interplay with celestia-node and sampling should be tested.

[evan-forbes]

During the onsite, we ran tests using our testground setup. We had trouble reaching consensus using 8MB blocks with 40 validators unless we allowed for 1GB/s connections for each validator. If we increased the proposal timeout for each node, we were able to reach consensus using 320MB/s connection per each node.

This appears to be caused by some inefficiencies in tendermint p2p stack, but we will be adding more introspection to the testground tests soon. We will continuously run various tests after the introspection is added, so I think its safe to close this issue.