Merge branch 'main' into rp/improve-single-node

.github/workflows/test.yml

@@ -41,7 +41,7 @@ jobs:
         run: make test-coverage
 
       - name: Upload coverage.txt
-        uses: codecov/[email protected].4
+        uses: codecov/[email protected].7
         with:
           file: ./coverage.txt
 

Makefile

@@ -28,16 +28,13 @@ PACKAGE_NAME          := github.com/celestiaorg/celestia-app/v3
 GOLANG_CROSS_VERSION  ?= v1.23.1
 # Set this to override the max square size of the binary
 OVERRIDE_MAX_SQUARE_SIZE ?=
-# Set this to override the upgrade height delay of the binary
-OVERRIDE_UPGRADE_HEIGHT_DELAY ?=
 
 # process linker flags
 ldflags = -X github.com/cosmos/cosmos-sdk/version.Name=celestia-app \
 		  -X github.com/cosmos/cosmos-sdk/version.AppName=celestia-appd \
 		  -X github.com/cosmos/cosmos-sdk/version.Version=$(VERSION) \
 		  -X github.com/cosmos/cosmos-sdk/version.Commit=$(COMMIT) \
-		  -X github.com/celestiaorg/celestia-app/v3/pkg/appconsts.OverrideSquareSizeUpperBoundStr=$(OVERRIDE_MAX_SQUARE_SIZE) \
-		  -X github.com/celestiaorg/celestia-app/v3/pkg/appconsts.OverrideUpgradeHeightDelayStr=$(OVERRIDE_UPGRADE_HEIGHT_DELAY)
+		  -X github.com/celestiaorg/celestia-app/v3/pkg/appconsts.OverrideSquareSizeUpperBoundStr=$(OVERRIDE_MAX_SQUARE_SIZE)
 
 BUILD_FLAGS := -tags "ledger" -ldflags '$(ldflags)'
 

app/benchmarks/README.md

@@ -6,7 +6,7 @@ This package contains benchmarks for the ABCI methods with the following transac
 - IBC update client
 - PayForBlobs
 
-## How to run
+## How to Run
 
 To run the benchmarks, run the following in the root directory:
 
@@ -20,8 +20,8 @@ The results are outlined in the [results](results.md) document.
 
 ## Key takeaways
 
-We decided to softly limit the number of messages contained in a block, via introducing the `MaxPFBMessages` and `MaxNonPFBMessages`, and checking against them in prepare proposal.
+We decided to softly limit the number of messages contained in a block, by introducing the `MaxPFBMessages` and `MaxNonPFBMessages`, and checking against them while preparing the proposal.
 
-This way, the default block construction mechanism will only propose blocks that respect these limitations. And if a block that doesn't respect them reached consensus, it will still be accepted since this rule is not consensus breaking.
+This way, the default block construction mechanism will only propose blocks that respect these limitations. And if a block that doesn't respect them reaches consensus, it will still be accepted since this rule is not consensus breaking.
 
-As specified in [results](results.md) document, those results were generated on 16 core 48GB RAM machine, and gave us certain thresholds. However, when we run the same experiments on the recommended validator setup, 4 cores 16GB RAM, the numbers were lower. These low numbers are what we used in the limits.
+As specified in the [results](results.md) document, those results were generated on a 16-core, 48GB RAM machine and gave us certain thresholds. However, when we ran the same experiments on the recommended validator setup, with a 4-core, 16GB RAM machine, the numbers were lower. These low numbers are what we used in the limits.

app/benchmarks/results.md

@@ -13,27 +13,27 @@ The benchmarks will be run using an in memory DB, then a local db, goleveldb.
 
 #### CheckTx
 
-A single `checkTx` of a `sendMsg` message takes 0.0003585 **ns** to execute. And it uses 74374 gas.
+A single `checkTx` of a `sendMsg` message takes 0.0003585 **ns** to execute and it uses 74374 gas.
 
-The transactions in an `8mb` block containing 31645 `sendMsg` messages take 6,29 s (6293858682 ns) to run `checkTx` on all of them. The total gas used is 1884371034 gas.
+The transactions in an `8MB` block containing 31645 `sendMsg` messages take 6,29 s (6293858682 ns) to run `checkTx` on all of them. The total gas used is 1884371034 gas.
 
 #### DeliverTx
 
-A single `deliverTx` of a `sendMsg` message takes 0.0002890 **ns** to execute. And it uses 103251 gas.
+A single `deliverTx` of a `sendMsg` message takes 0.0002890 **ns** to execute and it uses 103251 gas.
 
-The transactions in an `8mb` block containing 31645 `sendMsg` messages take 7,56 s (7564111078 ns) to run `deliverTx` on all of them. The total gas used is 2801272121 gas.
+The transactions in an `8MB` block containing 31645 `sendMsg` messages take 7,56 s (7564111078 ns) to run `deliverTx` on all of them. The total gas used is 2801272121 gas.
 
 #### PrepareProposal
 
-A single `prepareProposal` of a `sendMsg` message takes 0.0002801 **ns** to execute. And it uses 101110 gas.
+A single `prepareProposal` of a `sendMsg` message takes 0.0002801 **ns** to execute and it uses 101110 gas.
 
-An `8mb` block containing 31645 `sendMsg` messages takes 5,04 s (5049140917 ns) to execute. The total gas used 1843040790 gas.
+An `8MB` block containing 31645 `sendMsg` messages takes 5,04 s (5049140917 ns) to execute. The total gas used is 1843040790 gas.
 
 #### ProcessProposal
 
-A single `processProposal` of a `sendMsg` message takes 0.0002313 **ns** to execute. And it uses 101110 gas.
+A single `processProposal` of a `sendMsg` message takes 0.0002313 **ns** to execute and it uses 101110 gas.
 
-An `8mb` block containing 31645 `sendMsg` messages takes 5,17 s (5179850250 ns) to execute. The total gas used 1,843,040,790 gas.
+An `8MB` block containing 31645 `sendMsg` messages takes 5,17 s (5179850250 ns) to execute. The total gas used 1,843,040,790 gas.
 
 For the processing time of a block full of `sendMsg`, we benchmarked how much time they take depending on the number of transactions, and we have the following results:
 
@@ -123,7 +123,7 @@ Benchmarks of `DeliverTx` for a single PFB with different sizes:
 
 #### PrepareProposal: `BenchmarkPrepareProposal_PFB_Multi`
 
-The benchmarks for `PrepareProposal` for 8mb blocks containing PFBs of different sizes:
+The benchmarks for `PrepareProposal` for 8MB blocks containing PFBs of different sizes:
 
 | Benchmark Name                                                         | Block Size (MB) | Number of Transactions | Prepare Proposal Time (s) | Total Gas Used  | Transaction Size (Bytes) | Transaction Size (MB) |
 |------------------------------------------------------------------------|-----------------|------------------------|---------------------------|-----------------|--------------------------|-----------------------|
@@ -147,7 +147,7 @@ The benchmarks for `PrepareProposal` for 8mb blocks containing PFBs of different
 
 #### ProcessProposal: `BenchmarkProcessProposal_PFB_Multi`
 
-The benchmarks for `ProcessProposal` for 8mb blocks containing PFBs of different sizes:
+The benchmarks for `ProcessProposal` for 8MB blocks containing PFBs of different sizes:
 
 | Benchmark Name                                                         | Block Size (MB) | Number of Transactions | Process Proposal Time (s) | Total Gas Used  | Transaction Size (Bytes) | Transaction Size (MB) |
 |------------------------------------------------------------------------|-----------------|------------------------|---------------------------|-----------------|--------------------------|-----------------------|
@@ -217,7 +217,7 @@ The benchmarks of executing `deliverTx` on a single transaction containing an IB
 
 #### PrepareProposal: `BenchmarkIBC_PrepareProposal_Update_Client_Multi`
 
-Benchmarks of an `8mb` containing the maximum number of IBC `UpdateClient` with different number of signatures:
+Benchmarks of an `8MB` containing the maximum number of IBC `UpdateClient` with different number of signatures:
 
 | Benchmark Name                                                                | Block Size (MB)  | Number of Transactions  | Number of Validators | Number of Verified Signatures | Prepare Proposal Time (s)   | Total Gas Used   | Transaction Size (Bytes)   | Transaction Size (MB)  |
 |-------------------------------------------------------------------------------|------------------|-------------------------|----------------------|-------------------------------|-----------------------------|------------------|----------------------------|------------------------|
@@ -238,7 +238,7 @@ Benchmarks of an `8mb` containing the maximum number of IBC `UpdateClient` with
 
 #### ProcessProposal: `BenchmarkIBC_ProcessProposal_Update_Client_Multi`
 
-Benchmarks of an `8mb` containing the maximum number of IBC `UpdateClient` with different number of signatures:
+Benchmarks of an `8MB` containing the maximum number of IBC `UpdateClient` with different number of signatures:
 
 | Benchmark Name                                                                | Block Size (MB) | Number of Transactions | Number of Validators | Number of Verified Signatures | Process Proposal Time (s) | Total Gas Used | Transaction Size (Bytes) | Transaction Size (MB) |
 |-------------------------------------------------------------------------------|-----------------|------------------------|----------------------|-------------------------------|---------------------------|----------------|--------------------------|-----------------------|
@@ -627,27 +627,27 @@ Benchmarks of an `8mb` containing the maximum number of IBC `UpdateClient` with
 
 #### CheckTx
 
-A single `checkTx` of a `sendMsg` message takes 0.0003071 **ns** to execute. And it uses 74374 gas.
+A single `checkTx` of a `sendMsg` message takes 0.0003071 **ns** to execute and it uses 74374 gas.
 
-The transactions in an `8mb` block containing 31645 `sendMsg` messages take 6,45 s (6455816060 ns) to run `checkTx` on all of them. The total gas used is 1884371034 gas.
+The transactions in an `8MB` block containing 31645 `sendMsg` messages take 6,45 s (6455816060 ns) to run `checkTx` on all of them. The total gas used is 1884371034 gas.
 
 #### DeliverTx
 
-A single `deliverTx` of a `sendMsg` message takes 0.0003948 **ns** to execute. And it uses 103251 gas.
+A single `deliverTx` of a `sendMsg` message takes 0.0003948 **ns** to execute and it uses 103251 gas.
 
-The transactions in an `8mb` block containing 31645 `sendMsg` messages take 7,50 s (7506830940 ns) to run `deliverTx` on all of them. The total gas used is 2801272121 gas.
+The transactions in an `8MB` block containing 31645 `sendMsg` messages take 7,50 s (7506830940 ns) to run `deliverTx` on all of them. The total gas used is 2801272121 gas.
 
 #### PrepareProposal
 
-A single `prepareProposal` of a `sendMsg` message takes 0.0003943 **ns** to execute. And it uses 101110 gas.
+A single `prepareProposal` of a `sendMsg` message takes 0.0003943 **ns** to execute and it uses 101110 gas.
 
-An `8mb` block containing 31645 `sendMsg` messages takes 5,2 s (5242159792 ns) to execute. The total gas used 1843040790 gas.
+An `8MB` block containing 31645 `sendMsg` messages takes 5,2 s (5242159792 ns) to execute. The total gas used 1843040790 gas.
 
 #### ProcessProposal
 
-A single `processProposal` of a `sendMsg` message takes 0.0003010 **ns** to execute. And it uses 101110 gas.
+A single `processProposal` of a `sendMsg` message takes 0.0003010 **ns** to execute and it uses 101110 gas.
 
-An `8mb` block containing 31645 `sendMsg` messages takes 5,21 s (5214205041 ns) to execute. The total gas used 1843040790 gas.
+An `8MB` block containing 31645 `sendMsg` messages takes 5,21 s (5214205041 ns) to execute. The total gas used 1843040790 gas.
 
 ### `PFB` benchmarks
 
@@ -701,7 +701,7 @@ Benchmarks of `DeliverTx` for a single PFB with different sizes:
 
 #### PrepareProposal: `BenchmarkPrepareProposal_PFB_Multi`
 
-The benchmarks for `PrepareProposal` for 8mb blocks containing PFBs of different sizes:
+The benchmarks for `PrepareProposal` for 8MB blocks containing PFBs of different sizes:
 
 | Benchmark Name                                                         | Block Size (MB) | Number of Transactions | Prepare Proposal Time (s) | Total Gas Used  | Transaction Size (Bytes) | Transaction Size (MB) |
 |------------------------------------------------------------------------|-----------------|------------------------|---------------------------|-----------------|--------------------------|-----------------------|
@@ -725,7 +725,7 @@ The benchmarks for `PrepareProposal` for 8mb blocks containing PFBs of different
 
 #### ProcessProposal: `BenchmarkProcessProposal_PFB_Multi`
 
-The benchmarks for `ProcessProposal` for 8mb blocks containing PFBs of different sizes:
+The benchmarks for `ProcessProposal` for 8MB blocks containing PFBs of different sizes:
 
 | Benchmark Name                                                         | Block Size (MB) | Number of Transactions | Process Proposal Time (s) | Total Gas Used  | Transaction Size (Bytes) | Transaction Size (MB) |
 |------------------------------------------------------------------------|-----------------|------------------------|---------------------------|-----------------|--------------------------|-----------------------|
@@ -795,7 +795,7 @@ The benchmarks of executing `deliverTx` on a single transaction containing an IB
 
 #### PrepareProposal: `BenchmarkIBC_PrepareProposal_Update_Client_Multi`
 
-Benchmarks of an `8mb` containing the maximum number of IBC `UpdateClient` with different number of signatures:
+Benchmarks of an `8MB` containing the maximum number of IBC `UpdateClient` with different number of signatures:
 
 | Benchmark Name                                                                | Block Size (MB) | Number of Transactions | Number of Validators | Number of Verified Signatures | Prepare Proposal Time (s) | Total Gas Used | Transaction Size (Bytes) | Transaction Size (MB) |
 |-------------------------------------------------------------------------------|-----------------|------------------------|----------------------|-------------------------------|---------------------------|----------------|--------------------------|-----------------------|
@@ -817,7 +817,7 @@ Benchmarks of an `8mb` containing the maximum number of IBC `UpdateClient` with
 
 #### ProcessProposal: `BenchmarkIBC_ProcessProposal_Update_Client_Multi`
 
-Benchmarks of an `8mb` containing the maximum number of IBC `UpdateClient` with different number of signatures:
+Benchmarks of an `8MB` containing the maximum number of IBC `UpdateClient` with different number of signatures:
 
 | Benchmark Name                                                                | Block Size (MB) | Number of Transactions | Number of Validators | Number of Verified Signatures | Process Proposal Time (s) | Total Gas Used | Transaction Size (Bytes) | Transaction Size (MB) |
 |-------------------------------------------------------------------------------|-----------------|------------------------|----------------------|-------------------------------|---------------------------|----------------|--------------------------|-----------------------|

app/extend_block.go

@@ -25,8 +25,17 @@ func ExtendBlock(data coretypes.Data, appVersion uint64) (*rsmt2d.ExtendedDataSq
 	return da.ExtendShares(share.ToBytes(dataSquare))
 }
 
-// EmptyBlock returns true if the given block data is considered empty by the
+// IsEmptyBlock returns true if the given block data is considered empty by the
 // application at a given version.
+//
+// Deprecated: Use IsEmptyBlockRef for better performance with large data structures.
 func IsEmptyBlock(data coretypes.Data, _ uint64) bool {
 	return len(data.Txs) == 0
 }
+
+// IsEmptyBlockRef returns true if the application considers the given block data
+// empty at a given version.
+// This method passes the block data by reference for improved performance.
+func IsEmptyBlockRef(data *coretypes.Data, _ uint64) bool {
+	return len(data.Txs) == 0
+}

app/test/integration_test.go

@@ -275,13 +275,24 @@ func ExtendBlockTest(t *testing.T, block *coretypes.Block) {
 	}
 }
 
-func (s *IntegrationTestSuite) TestEmptyBlock() {
+func (s *IntegrationTestSuite) TestIsEmptyBlock() {
 	t := s.T()
 	emptyHeights := []int64{1, 2, 3}
 	for _, h := range emptyHeights {
 		blockRes, err := s.cctx.Client.Block(s.cctx.GoContext(), &h)
 		require.NoError(t, err)
-		require.True(t, app.IsEmptyBlock(blockRes.Block.Data, blockRes.Block.Header.Version.App))
+		require.True(t, app.IsEmptyBlock(blockRes.Block.Data, blockRes.Block.Header.Version.App)) //nolint:staticcheck
+		ExtendBlockTest(t, blockRes.Block)
+	}
+}
+
+func (s *IntegrationTestSuite) TestIsEmptyBlockRef() {
+	t := s.T()
+	emptyHeights := []int64{1, 2, 3}
+	for _, h := range emptyHeights {
+		blockRes, err := s.cctx.Client.Block(s.cctx.GoContext(), &h)
+		require.NoError(t, err)
+		require.True(t, app.IsEmptyBlockRef(&blockRes.Block.Data, blockRes.Block.Header.Version.App))
 		ExtendBlockTest(t, blockRes.Block)
 	}
 }

app/test/upgrade_test.go

@@ -38,9 +38,6 @@ func TestAppUpgradeV3(t *testing.T) {
 		t.Skip("skipping TestAppUpgradeV3 in short mode")
 	}
 
-	appconsts.OverrideUpgradeHeightDelayStr = "1"
-	defer func() { appconsts.OverrideUpgradeHeightDelayStr = "" }()
-
 	testApp, genesis := SetupTestAppWithUpgradeHeight(t, 3)
 	upgradeFromV1ToV2(t, testApp)
 
@@ -104,7 +101,6 @@ func TestAppUpgradeV3(t *testing.T) {
 	require.NoError(t, err)
 	require.Equal(t, v3.Version, getUpgradeResp.Upgrade.AppVersion)
 
-	// brace yourselfs, this part may take a while
 	initialHeight := int64(4)
 	for height := initialHeight; height < initialHeight+appconsts.UpgradeHeightDelay(testApp.GetChainID(), v2.Version); height++ {
 		appVersion := v2.Version
@@ -259,6 +255,7 @@ func SetupTestAppWithUpgradeHeight(t *testing.T, upgradeHeight int64) (*app.App,
 	encCfg := encoding.MakeConfig(app.ModuleEncodingRegisters...)
 	testApp := app.New(log.NewNopLogger(), db, nil, 0, encCfg, upgradeHeight, util.EmptyAppOptions{})
 	genesis := genesis.NewDefaultGenesis().
+		WithChainID("test").
 		WithValidators(genesis.NewDefaultValidator(testnode.DefaultValidatorAccountName)).
 		WithConsensusParams(app.DefaultInitialConsensusParams())
 	genDoc, err := genesis.Export()

cmd/celestia-appd/cmd/start.go

@@ -9,7 +9,6 @@ import (
 	"net"
 	"net/http"
 	"os"
-	"os/exec"
 	"path/filepath"
 	"runtime/pprof"
 	"strings"
@@ -580,6 +579,8 @@ sudo sysctl -p
 
 Then verify BBR is enabled:
 sysctl net.ipv4.tcp_congestion_control
+or
+cat /proc/sys/net/ipv4/tcp_congestion_control
 
 This node will get worse p2p performance using a different congestion control algorithm.
 If you need to bypass this check use the --force-no-bbr flag.
@@ -594,16 +595,15 @@ If you need to bypass this check use the --force-no-bbr flag.
 		return nil
 	}
 
-	cmd := exec.Command("sysctl", "net.ipv4.tcp_congestion_control")
-	output, err := cmd.Output()
+	file, err := os.ReadFile("/proc/sys/net/ipv4/tcp_congestion_control")
 	if err != nil {
 		fmt.Print(warning)
-		return fmt.Errorf("failed to execute 'sysctl net.ipv4.tcp_congestion_control' %w", err)
+		return fmt.Errorf("failed to read file '/proc/sys/net/ipv4/tcp_congestion_control' %w", err)
 	}
 
-	if !strings.Contains(string(output), "bbr") {
+	if !strings.Contains(string(file), "bbr") {
 		fmt.Print(warning)
-		return fmt.Errorf("BBR not enabled because output %v does not contain 'bbr'", string(output))
+		return fmt.Errorf("BBR not enabled because output %v does not contain 'bbr'", string(file))
 	}
 
 	return nil

docs/architecture/adr-015-namespace-id-size.md

@@ -50,7 +50,7 @@ Users will specify a version (1 byte) and a ID (28 bytes) in their PFB. Addition
 
 ### Criteria 1
 
-The namespace ID must provide at least 72 bits of randomness [^2] to satisfy criteria 1. Since an 8 byte namespace ID can only provide 64 bits of randomness, it fail to meet this criteria.
+The namespace ID must provide at least 72 bits of randomness to satisfy criteria 1. Since an 8 byte namespace ID can only provide 64 bits of randomness, it fail to meet this criteria.
 
 | Namespace ID size (bytes) | Criteria 1 |
 |---------------------------|------------|
@@ -229,7 +229,6 @@ Note: to verify the number of SHA256 compression invocations, we analyzed the nu
 - <https://github.com/celestiaorg/celestia-app/issues/1308>
 
 [^1]: This assumes a user uses sufficient entropy to generate the namespace ID and isn't front-run by an adversary prior to actually using the namespace.
-[^2]: <https://eager.io/blog/how-long-does-an-id-need-to-be/>
 [^3]: <https://kevingal.com/apps/collision.html>
 [^4]: <https://www.johndcook.com/blog/2017/01/10/probability-of-secure-hash-collisions/>
 <!-- markdown-link-check-disable -->