Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Lazy share writers #637

Merged
merged 73 commits into from
Apr 25, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
fad121c
abci: PrepareProposal (#6544)
tac0turtle Jul 27, 2021
364a56b
regenerate mocks, proto, mod/sum, and clean up remaining preprocesstxs
evan-forbes Feb 16, 2022
02d76fc
fix tests and revert to old go.mod
evan-forbes Feb 16, 2022
f291015
mockery
mconcat Oct 9, 2021
9b4699d
add processproposal proto/boilerplate/logic
mconcat Oct 8, 2021
d89dd59
gofmt
mconcat Oct 9, 2021
055d735
fix test
mconcat Oct 11, 2021
3266701
move UNKNOWN response behaviour to reject
mconcat Nov 19, 2021
a41d54b
fix test and add testing util code
evan-forbes Feb 16, 2022
5316340
pass full block data when proposing or processing proposals
evan-forbes Feb 16, 2022
6296291
linter
evan-forbes Feb 16, 2022
44c81db
add the process proposal method to the e2e app
evan-forbes Feb 16, 2022
65cdd2f
add missing kvstore abci method
evan-forbes Feb 16, 2022
5b2b444
pass block data and results for bass app
evan-forbes Feb 16, 2022
373ca76
use correct kvstore process logic for kvstore app
evan-forbes Feb 16, 2022
68e4e18
add new lazy share writers
evan-forbes Feb 24, 2022
29155fa
linter
evan-forbes Feb 24, 2022
4b36e76
remove unused arg
evan-forbes Feb 25, 2022
9bedec1
sort messages before exporting
evan-forbes Feb 26, 2022
8a568c2
formatting and bug fix
evan-forbes Feb 28, 2022
3c1d4fc
fix tests
evan-forbes Mar 1, 2022
e0e5198
allow for picking of square size when computing shares for the data s…
evan-forbes Mar 1, 2022
ec4810f
Merge branch 'v0.34.x-celestia' into evan/lazy-square-generation
evan-forbes Mar 31, 2022
0bc5297
remove accidental code duplication
evan-forbes Mar 31, 2022
7d94a5f
fix test from using wrong formatting directive
evan-forbes Mar 31, 2022
ad930c6
linter
evan-forbes Mar 31, 2022
8be0938
ci: backport lint configuration changes (#7225)
tychoish Nov 3, 2021
87098a2
lint: cleanup pending lint errors (#7237)
tychoish Nov 4, 2021
ca95b87
linter
evan-forbes Feb 24, 2022
7223835
remove unused arg
evan-forbes Feb 25, 2022
735e6c3
sort messages before exporting
evan-forbes Feb 26, 2022
1d87bda
formatting and bug fix
evan-forbes Feb 28, 2022
3f079e4
fix tests
evan-forbes Mar 1, 2022
6997a05
allow for picking of square size when computing shares for the data s…
evan-forbes Mar 1, 2022
8730f8c
fix test from using wrong formatting directive
evan-forbes Mar 31, 2022
ae74dea
linter
evan-forbes Mar 31, 2022
3517804
ci: backport lint configuration changes (#7225)
tychoish Nov 3, 2021
5d80670
lint: cleanup pending lint errors (#7237)
tychoish Nov 4, 2021
723a284
add new lazy share writers
evan-forbes Feb 24, 2022
812ed57
fix rebase
evan-forbes Apr 1, 2022
fd1d349
linter
evan-forbes Apr 1, 2022
0f76b4d
try ci with go 1.17
evan-forbes Apr 1, 2022
80ff2f6
Revert "try ci with go 1.17"
evan-forbes Apr 1, 2022
2a5a304
please work, linter gods
evan-forbes Apr 1, 2022
7599192
spelling
evan-forbes Apr 4, 2022
dbe113b
Merge branch 'evan/lazy-square-generation' of github.com:celestiaorg/…
evan-forbes Apr 4, 2022
6ca2779
initialize pending share using the const share size for capacity
evan-forbes Apr 4, 2022
4b7040a
force the last reserve bytes to be zero
evan-forbes Apr 8, 2022
760502e
remove todo
evan-forbes Apr 10, 2022
4aa56e0
add typecheck back to golang linter
evan-forbes Apr 10, 2022
0a8f6b4
Revert "ci: backport lint configuration changes (#7225)"
evan-forbes Apr 10, 2022
c949e96
Revert "lint: cleanup pending lint errors (#7237)"
evan-forbes Apr 10, 2022
fac4d89
Merge branch 'v0.34.x-celestia' into evan/lazy-square-generation
evan-forbes Apr 13, 2022
454e69f
Merge branch 'v0.34.x-celestia' into evan/lazy-square-generation
evan-forbes Apr 13, 2022
7850931
Merge branch 'v0.34.x-celestia' into evan/lazy-square-generation
evan-forbes Apr 13, 2022
c77506b
add the link to issue back in
evan-forbes Apr 13, 2022
7bb534f
removed unfinished comment
evan-forbes Apr 13, 2022
687486d
Merge branch 'v0.34.x-celestia' into evan/lazy-square-generation
evan-forbes Apr 18, 2022
7d2505d
switch fuzzer back to one minute
evan-forbes Apr 18, 2022
81fff48
regenerate proto
evan-forbes Apr 18, 2022
246b74e
better docs
evan-forbes Apr 18, 2022
7999ba9
fix encoding check to include the hash added to Data
evan-forbes Apr 18, 2022
6055a4f
add docs to CotiguousShareWriter
evan-forbes Apr 18, 2022
4161214
fix encoding check
evan-forbes Apr 18, 2022
8e54103
explain why the share reserve byte is zero
evan-forbes Apr 18, 2022
20e3b9c
use punctuation
evan-forbes Apr 23, 2022
3f9bfff
use punctuation
evan-forbes Apr 23, 2022
d7083a6
use punctuation
evan-forbes Apr 23, 2022
24bf826
use punctuation
evan-forbes Apr 23, 2022
d2f5252
use punctuation
evan-forbes Apr 23, 2022
e83377e
use punctuation
evan-forbes Apr 23, 2022
483f840
use clearer wording for compute shares docs
evan-forbes Apr 24, 2022
95de074
more accurate docs
evan-forbes Apr 24, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ linters:
- lll
- misspell
# - maligned
- misspell
- nakedret
- prealloc
# - scopelint
Expand Down
2 changes: 1 addition & 1 deletion blockchain/msgs_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ func TestBlockchainMessageVectors(t *testing.T) {
BlockRequest: &bcproto.BlockRequest{Height: math.MaxInt64}}},
"0a0a08ffffffffffffffff7f"},
{"BlockResponseMessage", &bcproto.Message{Sum: &bcproto.Message_BlockResponse{
BlockResponse: &bcproto.BlockResponse{Block: bpb}}}, "1a740a720a5b0a02080b1803220b088092b8c398feffffff012a0212003a20269ece38583f42aaf53fdd3abe1f570ab9b0d08284d080900966040a29df504c6a20e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b85512130a0b48656c6c6f20576f726c641a0022002801"},
BlockResponse: &bcproto.BlockResponse{Block: bpb}}}, "1a97010a94010a5b0a02080b1803220b088092b8c398feffffff012a0212003a20269ece38583f42aaf53fdd3abe1f570ab9b0d08284d080900966040a29df504c6a20e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b85512350a0b48656c6c6f20576f726c641a00220028013220269ece38583f42aaf53fdd3abe1f570ab9b0d08284d080900966040a29df504c"},
{"NoBlockResponseMessage", &bcproto.Message{Sum: &bcproto.Message_NoBlockResponse{
NoBlockResponse: &bcproto.NoBlockResponse{Height: 1}}}, "12020801"},
{"NoBlockResponseMessage", &bcproto.Message{Sum: &bcproto.Message_NoBlockResponse{
Expand Down
5 changes: 1 addition & 4 deletions libs/protoio/io_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,7 @@ func iotest(writer protoio.WriteCloser, reader protoio.ReadCloser) error {
if i != size {
panic("not enough messages read")
}
if err := reader.Close(); err != nil {
return err
}
return nil
return reader.Close()
}

type buffer struct {
Expand Down
10 changes: 2 additions & 8 deletions node/node.go
Original file line number Diff line number Diff line change
Expand Up @@ -1046,11 +1046,8 @@ func (n *Node) ConfigureRPC() error {

Config: *n.config.RPC,
})
if err := rpccore.InitGenesisChunks(); err != nil {
return err
}

return nil
return rpccore.InitGenesisChunks()
}

func (n *Node) startRPC() ([]net.Listener, error) {
Expand Down Expand Up @@ -1395,11 +1392,8 @@ func saveGenesisDoc(db dbm.DB, genDoc *types.GenesisDoc) error {
if err != nil {
return fmt.Errorf("failed to save genesis doc due to marshaling error: %w", err)
}
if err := db.SetSync(genesisDocKey, b); err != nil {
return err
}

return nil
return db.SetSync(genesisDocKey, b)
}

func createAndStartPrivValidatorSocketClient(
Expand Down
11 changes: 8 additions & 3 deletions pkg/prove/proof_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ func TestTxInclusion(t *testing.T) {
}

// compute the data availability header
shares, _ := typicalBlockData.ComputeShares()
shares, _, err := typicalBlockData.ComputeShares(0)
require.NoError(t, err)

squareSize := uint(math.Sqrt(float64(len(shares))))

Expand Down Expand Up @@ -101,7 +102,9 @@ func Test_genRowShares(t *testing.T) {
Messages: generateRandomlySizedMessages(10, 1000),
}

allShares, _ := typicalBlockData.ComputeShares()
// compute the data availability header
allShares, _, err := typicalBlockData.ComputeShares(0)
require.NoError(t, err)
rawShares := allShares.RawShares()

originalSquareSize := uint(math.Sqrt(float64(len(rawShares))))
Expand Down Expand Up @@ -137,7 +140,9 @@ func Test_genOrigRowShares(t *testing.T) {
Messages: generateRandomlySizedMessages(10, 1500),
}

allShares, _ := typicalBlockData.ComputeShares()
// compute the data availability header
allShares, _, err := typicalBlockData.ComputeShares(0)
require.NoError(t, err)
rawShares := allShares.RawShares()

genShares := genOrigRowShares(typicalBlockData, 8, 0, 7)
Expand Down
14 changes: 7 additions & 7 deletions proto/tendermint/abci/types.proto
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,8 @@ message RequestPrepareProposal {
}

message RequestProcessProposal {
tendermint.types.Header header = 1 [(gogoproto.nullable) = false];
tendermint.types.Data block_data = 2;
tendermint.types.Header header = 1 [(gogoproto.nullable) = false];
tendermint.types.Data block_data = 2;
}

//----------------------------------------
Expand All @@ -163,7 +163,7 @@ message Response {
ResponseLoadSnapshotChunk load_snapshot_chunk = 15;
ResponseApplySnapshotChunk apply_snapshot_chunk = 16;
ResponsePrepareProposal prepare_proposal = 17;
ResponseProcessProposal process_proposal = 18;
ResponseProcessProposal process_proposal = 18;
}
}

Expand Down Expand Up @@ -299,13 +299,13 @@ message ResponsePrepareProposal {
}

message ResponseProcessProposal {
Result result = 1;
Result result = 1;
repeated bytes evidence = 2;

enum Result {
UNKNOWN = 0; // Unknown result, invalidate
ACCEPT = 1; // proposal verified, vote on the proposal
REJECT = 2; // proposal invalidated
UNKNOWN = 0; // Unknown result, invalidate
ACCEPT = 1; // proposal verified, vote on the proposal
REJECT = 2; // proposal invalidated
}
}

Expand Down
Loading