NEP-507: Switching to Post State Root

neps/nep-0507.md

@@ -0,0 +1,119 @@
+---
+NEP: 507
+Title: Switching to Post State Root
+Authors: Robin Cheng, Anton Puhach, Alex Logunov, Yoon Hong 
+Status: Draft
+DiscussionsTo: https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit?usp=sharing
+Type: Protocol
+Version: 1.0.0
+Created: 2023-09-17
+LastUpdated: 2023-09-27
+---
+
+## Summary
+
+The NEP is to propose a chunk to contain post state root instead of pre state root.
+
+### Key terminology
+* Pre state root:
+  * Conceptually, chunk producers include transactions and receipts in a chunk *before* executing them.
+  * A block logically contains information about the state of the blockchain *before* the block, as well as how to modify the state during this block.
+  * As a matter of implementation, a chunk includes outgoing receipts from the previous chunk of the same shard. To fetch incoming receipts for execution, receipts from all chunks of the *same* block are needed.
+  * From a consensus or security perspective, a block only certifies the state *before* the block, not after.
+
+* Post state root:
+  * Conceptually, chunk producers include transactions and receipts in a chunk *after* executing them.
+  * A block logically contains information about the state of the blockchain *after* the block, as well as how the state transition was modified to arrive at this new state.
+  * As a matter of implementation, a chunk includes outgoing receipts produced by executing the txs and incoming receipts of the current chunk. To fetch incoming receipts for execution, receipts from all chunks of the *previous* block are needed.
+  * From a consensus or security perspective, a block certifies the state *after* the block.
+
+## Motivation
+The ideation of the project started as a pre-requsite for Stateless validation, which will be discussed in another NEP. 
+
+With stateless validation, we can no longer propose a chunk without executing it. A chunk proposal itself contains the state witness (proof of original state, plus the new state root), which could only be produced by executing the transactions and receipts in the chunk. This, along with other considerations, makes Post-State-Root a prerequisite.
+
+Having said that, it is worth noting that post state root iself can yield several critical benefits even without stateless validation and they are listed in the Benefits section below as well as in the reference doc.
+
+## Specification
+* The block structure mostly doesn’t change, except that it includes the hash of the post-state-roots from the chunks, as opposed to the hash of their pre-state-roots. Block production does not need to change; it will still be produced as soon as chunks are available.
+* There is a current optimization that everybody (including validators) executes a chunk in parallel after it is proposed, which is no longer possible since execution must now happen before proposal. To achieve a comparable optimization, the next chunk producer shall start executing the next chunk (in preparation for the next chunk's proposal) as soon as the current block is received, while the validators are validating the current chunk.
+* A chunk now includes outgoing receipts from the current chunk, rather than of the previous chunk.
+  * The outgoing receipts are still organized by target shard, and merklized like today. The only change is that these receipts are included in the current chunk, not the next chunk.
+* To execute a chunk (as a validator, chunk producer, or any other role), one needs transactions and incoming receipts. For transactions, nothing changes. For incoming receipts, we now need to fetch the partial outgoing receipts from the chunks of the *previous* block.
+* Chunk execution no longer depends on the current block. This is necessary to allow chunks to be produced at all. However, this makes the VRF (random seed, calculated when producing a block) one block behind. This is OK because transactions can only cause a contract execution one block later (except same-account transactions, which argubly cannot take advantage of known VRFs), so there's no way for someone to time a transaction until when the VRF is favorable. To prevent possible delay attacks (validators choosing not to produce a block, or user flooding the network to delay execution to the next block) after the transaction is submitted, we force the VRF used for a receipt to be the VRF of the block that the receipt was first produced in, so that the execution outcome of a receipt is deterministic as soon as it is produced.
+* ChunkExtra column is no longer needed after the transition to post-state-root is complete.
+
+## Reference Implementation
+**TBD**. Draft PR pending.
+
+[This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that:
+
+- Its interaction with other features is clear.
+- Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.)
+- It is reasonably clear how the feature would be implemented.
+- Corner cases are dissected by example.
+- For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges.
+
+The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.]
+
+## Security Implications
+So far, we have identified potential concern around 'delayed VRF'. See the description in the the Specification section. Some questions warrant additional scrutiny:
+* Is it a sound argument that local receipts (produced by transactions whose sender equals the receiver) cannot meaningfully exploit known VRFs? The argument is that if the sender/receiver is a wallet, then there is no contract code executed so the VRF is irrelevant. If the sender/receiver is a contract, the contract is sending a transaction on its own behalf, but at that point the contract had full control anyway so there's no need to rely on a specific VRF; it could just do whatever it wants to do, e.g. deploy a different contract that pretends the VRF was a certain result.
+* To prevent delay attacks after a transaction is already in the pipeline, we propose to have a receipt's VRF state be determined by the same block that the receipt is emitted in. Here we consider two scenarios:
+  * A chunk is proposed that includes a receipt that a party A cares about. A happens to operate a block producer, and upon seeing that the receipt was executed with an unfavorable VRF, chooses not to include the chunk in the block it produces. Next time we produce a new block, the same chunk might be included but with a different VRF. To prevent this, we ensure that the execution of a receipt always uses the same VRF state no matter when the execution happens.
+  * A user submits a transaction that will ultimately query the VRF. When a block is produced with a VRF that is unfavorable to the user, instead of letting the transaction continue to execute a receipt, the user floods the network with dummy transactions to fill the next chunk so that the receipt is pushed to the delayed receipts queue, causing it to be executed at a later chunk, when the VRF is more favorable. By ensuring that the VRF state is deterministic for each receipt, this attack is also prevented.
+  * However, we need to consider whether this is feasible to implement, and also scrutinize whether having a receipt whose VRF state is known leads to other security implications.
+
+## Alternatives
+One alternative considered was to implement stateless validation without post-state-root, by doing a whole round of stateless validation after the chunk is produced. This unfortunately cannot be made to work. See here for a discussion: https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit#heading=h.g4jghia40hcz
+
+## Future possibilities
+This NEP facilitates the implementation of Stateless Validation.
+
+## Consequences
+[Note] Please refer to [Post-State-Root changes](https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit#heading=h.1zjpu0g8edny) section in the reference doc for more details.
+
+### Positive
+* Enables Stateless Validation to be implementable.
+* The confusing and unintuitive behavior of encoding outgoing receipts of the previous chunk is removed.
+* Handling missed chunks is simpler.
+* The ChunkExtra column is removed.
+* Resharding no longer needs to take care of resharding outgoing receipts.
+
+### Negative
+* Contract executions can only use VRF from the previous block. This is a weaker security guarantee. See the Security Implications section for more details.
+
+### Backwards Compatibility
+As the proposal dramatically changes the way state is stored in each chunk and execution timing of a chunk, there is no easy way to support replayabililty unless we want to keep pre state root code forever. This naturally raises the question of limited replayability at nearcore level. Please refer to [How to deleted old code](https://docs.google.com/document/d/1ey2EKK6ccoivvI9iBFCUiL7wqqr8kkoMYCqFL3Rs67I/edit?usp=sharing) for more details. 
+
+## Unresolved Issues (Optional)
+* Detailed planning for pre state root to post state root Protocol upgrade: please refer to [the relevant section](https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit#heading=h.qt474ok5fh46)
+* Stateless validation is out of scope for this NEP and will be discussed separately.
+* Supporting nearcore level limited replayability to cope with Backward compatibility issue mentioned above.
+
+## Changelog
+
+### 1.0.0 - Initial Version
+
+> Placeholder for the context about when and who approved this NEP version.
+
+#### Benefits
+
+> List of benefits filled by the Subject Matter Experts while reviewing this version:
+
+- Benefit 1
+- Benefit 2
+
+#### Concerns
+
+> Template for Subject Matter Experts review for this version:
+> Status: New | Ongoing | Resolved
+
+|   # | Concern | Resolution | Status |
+| --: | :------ | :--------- | -----: |
+|   1 |         |            |        |
+|   2 |         |            |        |
+
+## Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).