wip

neps/nep-0507.md

@@ -0,0 +1,126 @@
+---
+NEP: 0507
+Title: Switching to Post State Root
+Authors: Robin Cheng, Anton Puhach, Alex Logunov, Yoon Hong 
+Status: Draft
+DiscussionsTo: https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit?usp=sharing
+Type: Protocol
+Version: 1.0.0
+Created: 2023-09-19
+LastUpdated: 2023-09-19
+---
+
+## Summary
+
+The NEP is to propose a chunk to contain post state root instead of pre state root.
+
+### Key terminology
+* Pre state root:
+  * Chunk producers include transactions and receipts in a chunk without executing them.
+  * A block only contains information about the state of the blockchain before the block, as well as how to modify the state during this block.
+  * A chunk includes outgoing receipts from the previous block of the same shard
+  * From a consensus or security perspective, a block only certifies the state before the block, not after.
+
+* Post state root:
+  * For incoming receipts, chunk producers fetch the outgoing receipts from the chunks of the previous block.
+  * Chunk includes outgoing receipts for the current height (after applying txn and incoming receipts)
+  * From a consensus or security perspective, a block certifies the state after the block.
+  * Execution of chunk in block N can happen in parallel with validation of block N.
+
+
+## Motivation
+The ideation of the project started as a pre-requsite for Stateless validation, which will be discussed in another NEP. 
+
+With stateless validation, we can no longer propose a chunk without executing it. The proposal itself is the state witness, which requires execution. Post-State-Root is therefore a prerequisite.
+
+Having said that, it is worth noting that post state root iself can yield several critical benefits even without stateless validation and they are listed in Benefits section below as well as in the reference doc.
+
+## Specification
+**TBD**
+[Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.]
+
+## Reference Implementation
+**TBD**
+
+[This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that:
+
+- Its interaction with other features is clear.
+- Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.)
+- It is reasonably clear how the feature would be implemented.
+- Corner cases are dissected by example.
+- For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges.
+
+The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.]
+
+## Security Implications
+So far, we have identified potential concern around 'delayed VRF'. 
+
+With Post-State-Root, we can only use the VRF from the previous block and will be is known beforehand to all chunk producers and other entities observing the blockchain.
+
+The situtation may not be that bad considering transactions cannot be immediately executed (unless it’s a local receipt); one gets no advantage to include a transaction when they see that the current VRF is favorable, as their transaction’s first receipt would only be executed with the next VRF.
+
+## Alternatives
+N/A
+[Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?]
+
+## Future possibilities
+**TBD** 
+
+## Consequences
+[Note] Please refer to [Post-State-Root changes](https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit#heading=h.1zjpu0g8edny) section in the reference doc for more details.
+
+### Positive
+* Retrieval of incoming receipts is much easier: simply go through outgoing receipts of the previous block and find the ones which has the target shard as a destination shard.
+* A chunk is executed before included in a block.
+* Missed Chunks are handled similarly but in the simplified manner.
+* State sync should no longer have the ChunkExtra issue.
+* The ChunkExtra column is removed.
+* Resharding no longer needs to take care of resharding outgoing receipts.
+
+### Neutral
+* The block structure mostly doesn’t change, except that it includes the hash of the post-state-roots from the chunks, as opposed to the hash of their pre-state-roots.
+* A chunk now includes outgoing receipts from the current chunk, rather than of the previous chunk.
+* For incoming receipts, we now need to fetch the partial outgoing receipts from the chunks of the previous block. 
+* State sync still requires incoming receipts for preceding missed chunks.
+* Resharding still needs to take into account previously missed chunks.
+
+### Negative
+* With Post-State-Root, we can only use the VRF from the previous block and will be is known beforehand to all chunk producers and other entities observing the blockchain.
+  * In Near, as transactions cannot be immediately executed (unless it’s a local receipt), one gets no advantage to include a transaction when they see that the current VRF is favorable, as their transaction’s first receipt would only be executed with the next VRF
+* Due to the delayed VRF, this leaves us with a potential issue of chunk producers deciding to not produce a block when they see an unfavorable VRF
+  * We can force all receipts to always execute with the VRF of the block they were generated from, so that even if they were included in a later chunk, they would have the same execution outcome.
+
+### Backwards Compatibility
+As the proposal dramatically changes the way state is stored in each chunk and execution timing of a chunk, there is no easy way to support replayabililty unless we want to keep pre state root code forever. This naturally raises the question of limited replayability at nearcore level. Please refer to [How to deleted old code](https://docs.google.com/document/d/1ey2EKK6ccoivvI9iBFCUiL7wqqr8kkoMYCqFL3Rs67I/edit?usp=sharing) for more details. 
+
+## Unresolved Issues (Optional)
+* Detailed planning for pre state root to post state root Protocol upgrade: please refer to [the relevant section](https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit#heading=h.qt474ok5fh46)
+* Stateless validation is out of scope for this NEP and will be discussed separately.
+* Supporting nearcore level limited replayability to cope with Backward compatibility issue mentioned above.
+
+## Changelog
+
+### 1.0.0 - Initial Version
+
+> Placeholder for the context about when and who approved this NEP version.
+
+#### Benefits
+
+> List of benefits filled by the Subject Matter Experts while reviewing this version:
+
+- Benefit 1
+- Benefit 2
+
+#### Concerns
+
+> Template for Subject Matter Experts review for this version:
+> Status: New | Ongoing | Resolved
+
+|   # | Concern | Resolution | Status |
+| --: | :------ | :--------- | -----: |
+|   1 |         |            |        |
+|   2 |         |            |        |
+
+## Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).