Tracking: Non-interactive defaults

[evan-forbes]

With our current message inclusion check, we are not checking the location of the messages, only that each message has a corresponding commitment in a MsgPayForData. This works for ensuring that each message was included, but this doesn't actually check that the commitment is actually a subtree root to one of the data availability header row or column roots.

In order to do this, we have to have block producers organize the message layout to follow the non-interactive defaults. This will involve a refactor during the app's PrepareProposal call, along with adding the message starting location to wrapped txs.

Additionally, we need validators and full nodes to check that each share commitment is actually a subtree root to one of the data availability headers. This will require a refactor to the app's ProcessProposal call, and is blocked by the functionality described in the above paragraph.

• Add MessageStartIndex field to the malleated transactions struct Use Wrapped Txs celestia-core#462
• Create a good API for checking for subtree root inclusion Create API to check for message inclusion #381
• Refactor PrepareProposal to organize the message layout according to the non-interactive defaults
• RefactorProcessProposal to check that subtree roots are actually part of data root #383

A huge bonus would be if we would also implement message inclusion fraud proofs, but this does not have to be completed to close this issue

[evan-forbes]

I have this listed as an incentivized testnet feature for now, as I don't think we will have it ready in time for testnet considering our other priorities, BUT we can (and should imho) obviously include this in an upgrade to testnet after we finish.

[evan-forbes]

The current plan for introducing features from the non-interactive defaults branch is roughly in this order

Wrapped Transactions

• feat: add share index to malleated (wrapped) txs celestia-core#819

Message inclusion

• feat: Cache and traverse nmt sub tree roots #549
• feat: add path generation for nmt subroots #621
• feat: finish msg inclusion api #681

Encoding

• chore: move share encoding logic from core -> app #627
• chore: refactor share splitting and merging #637
• feat: implement non-interactive defaults logic #680

Modify ABCI++ methods

• feat!: Refactor PrepareProposal to produce blocks using the non-interactive defaults #692
• RefactorProcessProposal to check that subtree roots are actually part of data root #383

Documentation

• ADR 006: Non-interactive Defaults, Wrapped Txs, and Subtree Root Message Inclusion Checks #673

notes: feature branch

ccing Reviewers: @rootulp @sweexordious @adlerjohn

[rootulp]

[Question] I discovered https://github.com/celestiaorg/celestia-core/blob/v0.34.x-celestia/pkg/prove/proof.go#L101-L109 which assumes that there is no padding between compact shares. Does TxInclusion in celestia-core also need to be updated to adhere to non-interactive-defaults behavior?

[evan-forbes]

I discovered https://github.com/celestiaorg/celestia-core/blob/v0.34.x-celestia/pkg/prove/proof.go#L101-L109 which assumes that there is no padding between compact shares. Does TxInclusion in celestia-core also need to be updated to adhere to non-interactive-defaults behavior?

whoops yeah great point, forgot to write this down. it definitely will, as creating proofs requires the entire row, which will likely include message shares.

as mentioned in the audit this morning by Sergio, we should look into how to use the blockstore in the app, we might just be able to move that endpoint there. It might slightly annoying to change in the sdk, but totally doable.

we could also move over the data commitment rpc endpoint if that's the case