From e8ab530dfc70ea6ea4331c5a5794601e8f6d24db Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 27 Oct 2020 11:22:53 -0400 Subject: [PATCH 01/51] Organize consensus constants alphabetically. --- specs/consensus.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 64e6303..4c9e2a8 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -30,18 +30,19 @@ Consensus Rules | name | type | value | unit | description | | ------------------------------------ | -------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `VERSION` | `uint64` | `1` | | Version of the LazyLedger chain. Breaking changes (hard forks) must update this parameter. | +| `AVAILABLE_DATA_ORIGINAL_SQUARE_MAX` | `uint64` | | `share` | Maximum number of rows/columns of the original data [shares](data_structures.md#share) in [square layout](data_structures.md#arranging-available-data-into-shares). | | `CHAIN_ID` | `uint64` | `1` | | Chain ID. Each chain assigns itself a (unique) ID. | +| `GENESIS_COIN_COUNT` | `uint64` | `10**8` | `4u` | `(= 100000000)` Number of coins at genesis. | +| `GRAFFITI_BYTES` | `uint64` | `32` | `byte` | Maximum size of transaction graffiti, in bytes. | +| `MAX_VALIDATORS` | `uint16` | `64` | | Maximum number of active validators. | | `NAMESPACE_ID_BYTES` | `uint64` | `8` | `byte` | Size of namespace ID, in bytes. | | `NAMESPACE_ID_MAX_RESERVED` | `uint64` | `255` | | Value of maximum reserved namespace ID (inclusive). 1 byte worth of IDs. | -| `SHARE_SIZE` | `uint64` | `256` | `byte` | Size of transaction and message [shares](data_structures.md#share), in bytes. | | `SHARE_RESERVED_BYTES` | `uint64` | `1` | `byte` | Bytes reserved at the beginning of each [share](data_structures.md#share). Must be sufficient to represent `SHARE_SIZE`. | -| `AVAILABLE_DATA_ORIGINAL_SQUARE_MAX` | `uint64` | | `share` | Maximum number of rows/columns of the original data [shares](data_structures.md#share) in [square layout](data_structures.md#arranging-available-data-into-shares). | -| `GENESIS_COIN_COUNT` | `uint64` | `10**8` | `4u` | `(= 100000000)` Number of coins at genesis. | -| `UNBONDING_DURATION` | `uint32` | | `block` | Duration, in blocks, for unbonding a validator or delegation. | -| `MAX_VALIDATORS` | `uint16` | `64` | | Maximum number of active validators. | +| `SHARE_SIZE` | `uint64` | `256` | `byte` | Size of transaction and message [shares](data_structures.md#share), in bytes. | | `STATE_SUBTREE_RESERVED_BYTES` | `uint64` | `1` | `byte` | Number of bytes reserved to identify state subtrees. | -| `GRAFFITI_BYTES` | `uint64` | `32` | `byte` | Maximum size of transaction graffiti, in bytes. | +| `UNBONDING_DURATION` | `uint32` | | `block` | Duration, in blocks, for unbonding a validator or delegation. | +| `VERSION` | `uint64` | `1` | | Version of the LazyLedger chain. Breaking changes (hard forks) must update this parameter. | + ### Reserved Namespace IDs From b9cb328c195d0179859091189552bb7434cc0fff Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 27 Oct 2020 15:46:46 -0400 Subject: [PATCH 02/51] Rename lastCommitRoot -> lastCommitHash. --- specs/data_structures.md | 2 +- specs/figures/block_data_structures.dot | 2 +- specs/figures/block_data_structures.svg | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/data_structures.md b/specs/data_structures.md index e39629a..2d54026 100644 --- a/specs/data_structures.md +++ b/specs/data_structures.md @@ -110,7 +110,7 @@ Block header, which is fully downloaded by both full clients and light clients. | `height` | [Height](#type-aliases) | Block height. The genesis block is at height `1`. | | `timestamp` | [Timestamp](#timestamp) | Timestamp of this block. | | `lastBlockID` | [BlockID](#blockid) | Previous block's ID. | -| `lastCommitRoot` | [HashDigest](#hashdigest) | Previous block's Tendermint commit root. | +| `lastCommitHash` | [HashDigest](#hashdigest) | Previous block's Tendermint commit hash. | | `consensusRoot` | [HashDigest](#hashdigest) | Merkle root of [consensus parameters](#consensus-parameters) for this block. | | `stateCommitment` | [HashDigest](#hashdigest) | The [state root](#state) after this block's transactions are applied. | | `availableDataOriginalSquareSize` | `uint64` | The number of rows/columns of the original data [shares](data_structures.md#share) in [square layout](data_structures.md#arranging-available-data-into-shares) for this block. Must be a power of 2. | diff --git a/specs/figures/block_data_structures.dot b/specs/figures/block_data_structures.dot index 296a7c6..0f6d375 100644 --- a/specs/figures/block_data_structures.dot +++ b/specs/figures/block_data_structures.dot @@ -22,7 +22,7 @@ digraph G { subgraph cluster_header { label = "header"; - struct1 [label = "height | timestamp | lastBlockID | lastCommitRoot | consensusRoot | stateCommitment | availableDataOriginalSquareSize | availableDataRoot | proposerAddress"]; + struct1 [label = "height | timestamp | lastBlockID | lastCommitHash | consensusRoot | stateCommitment | availableDataOriginalSquareSize | availableDataRoot | proposerAddress"]; } } diff --git a/specs/figures/block_data_structures.svg b/specs/figures/block_data_structures.svg index f139972..61aaf40 100644 --- a/specs/figures/block_data_structures.svg +++ b/specs/figures/block_data_structures.svg @@ -78,7 +78,7 @@ lastBlockID -lastCommitRoot +lastCommitHash consensusRoot From 96d7173fe6cc51173c24744d000842529e0e45b5 Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 28 Oct 2020 12:56:12 -0400 Subject: [PATCH 03/51] Add basic rules for header validity. --- specs/consensus.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/specs/consensus.md b/specs/consensus.md index 4c9e2a8..ff99e5f 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -10,6 +10,8 @@ Consensus Rules - [Leader Selection](#leader-selection) - [Fork Choice](#fork-choice) - [Block Validity](#block-validity) + - [Block Structure](#block-structure) + - [Header](#header) - [State Transitions](#state-transitions) - [Validators and Delegations](#validators-and-delegations) - [Formatting](#formatting) @@ -76,6 +78,27 @@ Consensus Rules ## Block Validity +This section specifies the entirety of the validity rules for a newly-seen block. + +### Block Structure + +Before executing [state transitions](#state-transitions), the structure of the [block](./data_structures.md#block) must be verified. The `header`, `availableDataHeader`, `availableData`, and `lastCommit` fields of the block are all downloaded or otherwise acquired. If they cannot all be acquired entirely, the block is ignored, but is not explicitly considered invalid by consensus rules. Further implications of ignoring a block are found in the [networking spec](./networking.md). + +#### Header + +The header is the first thing that is downloaded from the new block, and commits to everything inside the block in some way. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: + +1. `header.height` == `prev.header.height + 1`. +1. `header.timestamp` > `prev.header.timestamp`. +1. `header.lastBlockID` == the [block ID](./data_structures.md#blockid) of `prev`. +1. `header.lastCommitHash` == the [hash](./data_structures.md#hashing) of `lastCommit`. +1. `header.consensusRoot` == the value computed [here](./data_structures.md#consensus-parameters). +1. `header.stateCommitment` == the root of the state, computed [with the application of all state transitions in this block](#state-transitions). +1. `header.availableDataOriginalSquareSize` <= [`AVAILABLE_DATA_ORIGINAL_SQUARE_MAX`](#constants). +1. `header.availableDataOriginalSquareSize` is a power of `2`. +1. `header.availableDataRoot` == TODO available data root +1. `header.proposerAddress` == the [leader](#leader-selection) for `header.height`. + ### State Transitions #### Validators and Delegations From fc1adaa95e4a9784ad3659da7922289c6fdfdf11 Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 3 Nov 2020 15:14:56 -0500 Subject: [PATCH 04/51] Clarify serialization with deterministic protobuf. --- specs/data_structures.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specs/data_structures.md b/specs/data_structures.md index 2d54026..6a764d7 100644 --- a/specs/data_structures.md +++ b/specs/data_structures.md @@ -195,7 +195,11 @@ Output of the [signing](#public-key-cryptography) process. ## Serialization -Objects that are committed to or signed over require a canonical serialization. This is done using TODO. +Objects that are committed to or signed over require a canonical serialization. This is done using a deterministic (and thus, bijective) variant of protobuf defined [here](https://github.com/cosmos/cosmos-sdk/blob/master/docs/architecture/adr-027-deterministic-protobuf-serialization.md). + +Note: there are two requirements for a serialization scheme, should this need to be changed: +1. Must be bijective. +1. Serialization must include the length of dynamic structures (e.g. arrays with variable length). ## Hashing From 09645e1054a81146454598830ffa89d236a7ba5e Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 3 Nov 2020 21:57:26 -0500 Subject: [PATCH 05/51] Add some info on parsing checks. --- specs/consensus.md | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index ff99e5f..8566300 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -11,7 +11,9 @@ Consensus Rules - [Fork Choice](#fork-choice) - [Block Validity](#block-validity) - [Block Structure](#block-structure) - - [Header](#header) + - [`header`](#header) + - [`availableDataHeader`](#availabledataheader) + - [`lastCommit`](#lastcommit) - [State Transitions](#state-transitions) - [Validators and Delegations](#validators-and-delegations) - [Formatting](#formatting) @@ -84,7 +86,12 @@ This section specifies the entirety of the validity rules for a newly-seen block Before executing [state transitions](#state-transitions), the structure of the [block](./data_structures.md#block) must be verified. The `header`, `availableDataHeader`, `availableData`, and `lastCommit` fields of the block are all downloaded or otherwise acquired. If they cannot all be acquired entirely, the block is ignored, but is not explicitly considered invalid by consensus rules. Further implications of ignoring a block are found in the [networking spec](./networking.md). -#### Header +The following fields are parsed (i.e. [deserialized](./data_structures.md#serialization)). If they cannot be parsed, the block is ignored but not explicitly invalid as above. +1. [header](./data_structures.md#header) +1. [availableDataHeader](./data_structures.md#availabledataheader) +1. [lastCommit](./data_structures.md#commit) + +#### `header` The header is the first thing that is downloaded from the new block, and commits to everything inside the block in some way. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: @@ -99,6 +106,16 @@ The header is the first thing that is downloaded from the new block, and commits 1. `header.availableDataRoot` == TODO available data root 1. `header.proposerAddress` == the [leader](#leader-selection) for `header.height`. +TODO define the genesis block + +#### `availableDataHeader` + +1. `availableDataHeader.` + +#### `lastCommit` + + + ### State Transitions #### Validators and Delegations From 530e254ec34f91f7ba7bd57edf60651aabed9c95 Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 4 Nov 2020 14:44:12 -0500 Subject: [PATCH 06/51] Split up logic around available data acquisition. --- specs/consensus.md | 8 +++++--- specs/networking.md | 14 ++++++++++++++ 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 8566300..1f98144 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -84,13 +84,15 @@ This section specifies the entirety of the validity rules for a newly-seen block ### Block Structure -Before executing [state transitions](#state-transitions), the structure of the [block](./data_structures.md#block) must be verified. The `header`, `availableDataHeader`, `availableData`, and `lastCommit` fields of the block are all downloaded or otherwise acquired. If they cannot all be acquired entirely, the block is ignored, but is not explicitly considered invalid by consensus rules. Further implications of ignoring a block are found in the [networking spec](./networking.md). +Before executing [state transitions](#state-transitions), the structure of the [block](./data_structures.md#block) must be verified. -The following fields are parsed (i.e. [deserialized](./data_structures.md#serialization)). If they cannot be parsed, the block is ignored but not explicitly invalid as above. +The following block fields are acquired from the network and parsed (i.e. [deserialized](./data_structures.md#serialization)). If they cannot be parsed, the block is ignored but is not explicitly considered invalid by consensus rules. Further implications of ignoring a block are found in the [networking spec](./networking.md). 1. [header](./data_structures.md#header) 1. [availableDataHeader](./data_structures.md#availabledataheader) 1. [lastCommit](./data_structures.md#commit) +If the above fields are parsed successfully, the available data `availableData` is acquired in erasure-coded form as [a list of share rows](./networking.md#availabledata), then parsed. If it cannot be parsed, the block is ignored but not explicitly invalid, as above. + #### `header` The header is the first thing that is downloaded from the new block, and commits to everything inside the block in some way. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: @@ -110,7 +112,7 @@ TODO define the genesis block #### `availableDataHeader` -1. `availableDataHeader.` +1. Length of `availableDataHeader.availableDataCommitments` #### `lastCommit` diff --git a/specs/networking.md b/specs/networking.md index 59ff6b7..9bd983d 100644 --- a/specs/networking.md +++ b/specs/networking.md @@ -2,5 +2,19 @@ Networking === - [Wire Format](#wire-format) + - [AvailableData](#availabledata) + - [AvailableDataRow](#availabledatarow) ## Wire Format + +### AvailableData + +| name | type | description | +| ------------------- | ----------------------------------------- | ------------- | +| `availableDataRows` | [AvailableDataRow](#availabledatarow)`[]` | List of rows. | + +### AvailableDataRow + +| name | type | description | +| -------- | --------------------------------------- | ---------------- | +| `shares` | [Share](./data_structures.md#share)`[]` | Shares in a row. | From e1b4399f4c9ab6cc262899c68c26b3bbc62ba918 Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 4 Nov 2020 14:47:12 -0500 Subject: [PATCH 07/51] Clean up headings and notation. --- specs/consensus.md | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 1f98144..a32036f 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -11,9 +11,10 @@ Consensus Rules - [Fork Choice](#fork-choice) - [Block Validity](#block-validity) - [Block Structure](#block-structure) - - [`header`](#header) - - [`availableDataHeader`](#availabledataheader) - - [`lastCommit`](#lastcommit) + - [`block.header`](#blockheader) + - [`block.availableDataHeader`](#blockavailabledataheader) + - [`block.lastCommit`](#blocklastcommit) + - [`block.availableData`](#blockavailabledata) - [State Transitions](#state-transitions) - [Validators and Delegations](#validators-and-delegations) - [Formatting](#formatting) @@ -80,22 +81,22 @@ Consensus Rules ## Block Validity -This section specifies the entirety of the validity rules for a newly-seen block. +This section specifies the entirety of the validity rules for a newly-seen block, `block`. ### Block Structure Before executing [state transitions](#state-transitions), the structure of the [block](./data_structures.md#block) must be verified. The following block fields are acquired from the network and parsed (i.e. [deserialized](./data_structures.md#serialization)). If they cannot be parsed, the block is ignored but is not explicitly considered invalid by consensus rules. Further implications of ignoring a block are found in the [networking spec](./networking.md). -1. [header](./data_structures.md#header) -1. [availableDataHeader](./data_structures.md#availabledataheader) -1. [lastCommit](./data_structures.md#commit) +1. [block.header](./data_structures.md#header) +1. [block.availableDataHeader](./data_structures.md#availabledataheader) +1. [block.lastCommit](./data_structures.md#commit) -If the above fields are parsed successfully, the available data `availableData` is acquired in erasure-coded form as [a list of share rows](./networking.md#availabledata), then parsed. If it cannot be parsed, the block is ignored but not explicitly invalid, as above. +If the above fields are parsed successfully, the available data `block.availableData` is acquired in erasure-coded form as [a list of share rows](./networking.md#availabledata), then parsed. If it cannot be parsed, the block is ignored but not explicitly invalid, as above. -#### `header` +#### `block.header` -The header is the first thing that is downloaded from the new block, and commits to everything inside the block in some way. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: +The block header `block.header` (`header` for short) is the first thing that is downloaded from the new block, and commits to everything inside the block in some way. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: 1. `header.height` == `prev.header.height + 1`. 1. `header.timestamp` > `prev.header.timestamp`. @@ -110,11 +111,15 @@ The header is the first thing that is downloaded from the new block, and commits TODO define the genesis block -#### `availableDataHeader` +#### `block.availableDataHeader` 1. Length of `availableDataHeader.availableDataCommitments` -#### `lastCommit` +#### `block.lastCommit` + + + +#### `block.availableData` From 5cd25ed2549fb1e2530218e173a73c18cffa503b Mon Sep 17 00:00:00 2001 From: John Adler Date: Thu, 5 Nov 2020 08:41:49 -0500 Subject: [PATCH 08/51] Add rules for available data header. --- specs/consensus.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/consensus.md b/specs/consensus.md index a32036f..65a7e8a 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -113,7 +113,9 @@ TODO define the genesis block #### `block.availableDataHeader` -1. Length of `availableDataHeader.availableDataCommitments` +The available data header `block.availableDataHeader` (`availableDataHeader` for short) is then downloaded. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. + +1. Length of `availableDataHeader.availableDataCommitments` == `2 * header.availableDataOriginalSquareSize`. #### `block.lastCommit` From d48d26293ab8ec7d15885f3ccba6303cc8ca329d Mon Sep 17 00:00:00 2001 From: John Adler Date: Thu, 5 Nov 2020 13:45:39 -0500 Subject: [PATCH 09/51] Clarify available data header rules. --- specs/consensus.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/consensus.md b/specs/consensus.md index 65a7e8a..81fece9 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -113,9 +113,10 @@ TODO define the genesis block #### `block.availableDataHeader` -The available data header `block.availableDataHeader` (`availableDataHeader` for short) is then downloaded. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. +The available data header `block.availableDataHeader` (`availableDataHeader` for short) is then processed. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. The following checks must be true: 1. Length of `availableDataHeader.availableDataCommitments` == `2 * header.availableDataOriginalSquareSize`. +1. The length of each element in `availableDataHeader.availableDataCommitments` must be [`32`](./consensus.md#hashing). #### `block.lastCommit` From 19fc54b7c709f7d572ad310a4f0c0506477fe6f1 Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 18:14:00 -0500 Subject: [PATCH 10/51] Add rules for last commit. --- specs/consensus.md | 13 +++++++++++-- specs/data_structures.md | 12 ++++++------ 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 81fece9..2d56cd7 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -96,7 +96,7 @@ If the above fields are parsed successfully, the available data `block.available #### `block.header` -The block header `block.header` (`header` for short) is the first thing that is downloaded from the new block, and commits to everything inside the block in some way. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: +The [block header](./data_structures.md#header) `block.header` (`header` for short) is the first thing that is downloaded from the new block, and commits to everything inside the block in some way. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: 1. `header.height` == `prev.header.height + 1`. 1. `header.timestamp` > `prev.header.timestamp`. @@ -113,14 +113,23 @@ TODO define the genesis block #### `block.availableDataHeader` -The available data header `block.availableDataHeader` (`availableDataHeader` for short) is then processed. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. The following checks must be true: +The [available data header](./data_structures.md#availabledataheader)) `block.availableDataHeader` (`availableDataHeader` for short) is then processed. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. The following checks must be true: 1. Length of `availableDataHeader.availableDataCommitments` == `2 * header.availableDataOriginalSquareSize`. 1. The length of each element in `availableDataHeader.availableDataCommitments` must be [`32`](./consensus.md#hashing). #### `block.lastCommit` +The last [commit](./data_structures.md#commit) `block.lastCommit` (`lastCommit` for short) is processed next. This is the Tendermint commit (i.e. polka of votes) _for the previous block_. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: +1. `lastCommit.height` == `prev.header.height`. +1. `lastCommit.round` >= `1`. +1. `lastCommit.blockID` == the [block ID](./data_structures.md#blockid) of `prev`. +1. Length of `lastCommit.signatures` <= [`MAX_VALIDATORS`](#constants). +1. Each of `lastCommit.signatures` must be a valid [CommitSig](./data_structures.md#commitsig) +1. The sum of the votes for `prev` in `lastCommit` must be at least 2/3 (rounded up) of the voting power of `prev`'s next validator set. + +TODO define specifically how to validate commitsigs #### `block.availableData` diff --git a/specs/data_structures.md b/specs/data_structures.md index 6a764d7..47c0196 100644 --- a/specs/data_structures.md +++ b/specs/data_structures.md @@ -136,12 +136,12 @@ Data that is [erasure-coded](#erasure-coding) for [data availability checks](htt ### Commit -| name | type | description | -| ------------ | --------------------------- | ----------- | -| `height` | [Height](#type-aliases) | | -| `round` | [Round](#type-aliases) | | -| `blockID` | [BlockID](#blockid) | | -| `signatures` | [CommitSig](#commitsig)`[]` | | +| name | type | description | +| ------------ | --------------------------- | ---------------------------------- | +| `height` | [Height](#type-aliases) | Block height. | +| `round` | [Round](#type-aliases) | Round. Incremented on view change. | +| `blockID` | [BlockID](#blockid) | Block ID of the previous block. | +| `signatures` | [CommitSig](#commitsig)`[]` | List of signatures. | ### Timestamp From 56ca555181075ad74a8eda3f65786209fbdc0fad Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 18:29:26 -0500 Subject: [PATCH 11/51] Clean up, add initial available data setup. --- specs/consensus.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 2d56cd7..9a45949 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -92,7 +92,7 @@ The following block fields are acquired from the network and parsed (i.e. [deser 1. [block.availableDataHeader](./data_structures.md#availabledataheader) 1. [block.lastCommit](./data_structures.md#commit) -If the above fields are parsed successfully, the available data `block.availableData` is acquired in erasure-coded form as [a list of share rows](./networking.md#availabledata), then parsed. If it cannot be parsed, the block is ignored but not explicitly invalid, as above. +If the above fields are parsed successfully, the available data `block.availableData` is acquired in erasure-coded form as [a list of share rows](./networking.md#arranging-available-data-into-shares), then parsed. If it cannot be parsed, the block is ignored but not explicitly invalid, as above. #### `block.header` @@ -113,14 +113,14 @@ TODO define the genesis block #### `block.availableDataHeader` -The [available data header](./data_structures.md#availabledataheader)) `block.availableDataHeader` (`availableDataHeader` for short) is then processed. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. The following checks must be true: +The [available data header](./data_structures.md#availabledataheader)) `block.availableDataHeader` (`availableDataHeader` for short) is then processed. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. The following checks must be `true`: 1. Length of `availableDataHeader.availableDataCommitments` == `2 * header.availableDataOriginalSquareSize`. 1. The length of each element in `availableDataHeader.availableDataCommitments` must be [`32`](./consensus.md#hashing). #### `block.lastCommit` -The last [commit](./data_structures.md#commit) `block.lastCommit` (`lastCommit` for short) is processed next. This is the Tendermint commit (i.e. polka of votes) _for the previous block_. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: +The last [commit](./data_structures.md#commit) `block.lastCommit` (`lastCommit` for short) is processed next. This is the Tendermint commit (i.e. polka of votes) _for the previous block_. For previous block `prev` and previous block header `prev.header`, the following checks must be `true`: 1. `lastCommit.height` == `prev.header.height`. 1. `lastCommit.round` >= `1`. @@ -133,7 +133,13 @@ TODO define specifically how to validate commitsigs #### `block.availableData` +The block's [available data](./data_structures.md#availabledata) (analogous to transactions in contemporary blockchain designs) `block.availableData` (`availableData` for short) is finally processed. The [list of share rows](./networking.md#arranging-available-data-into-shares) is parsed into actual data structures; if parsing fails here, the block is invalid. +TODO define fraud proof for invalid block here + +Once parsed, the following checks must be `true`: + +1. ### State Transitions From f8c51f211f701c9bd9eb7723fc693948920e8241 Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 18:49:16 -0500 Subject: [PATCH 12/51] Fix bad links. --- specs/consensus.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 9a45949..ea25339 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -92,7 +92,7 @@ The following block fields are acquired from the network and parsed (i.e. [deser 1. [block.availableDataHeader](./data_structures.md#availabledataheader) 1. [block.lastCommit](./data_structures.md#commit) -If the above fields are parsed successfully, the available data `block.availableData` is acquired in erasure-coded form as [a list of share rows](./networking.md#arranging-available-data-into-shares), then parsed. If it cannot be parsed, the block is ignored but not explicitly invalid, as above. +If the above fields are parsed successfully, the available data `block.availableData` is acquired in erasure-coded form as [a list of share rows](./networking.md#availabledata), then parsed. If it cannot be parsed, the block is ignored but not explicitly invalid, as above. #### `block.header` @@ -133,7 +133,7 @@ TODO define specifically how to validate commitsigs #### `block.availableData` -The block's [available data](./data_structures.md#availabledata) (analogous to transactions in contemporary blockchain designs) `block.availableData` (`availableData` for short) is finally processed. The [list of share rows](./networking.md#arranging-available-data-into-shares) is parsed into actual data structures; if parsing fails here, the block is invalid. +The block's [available data](./data_structures.md#availabledata) (analogous to transactions in contemporary blockchain designs) `block.availableData` (`availableData` for short) is finally processed. The [list of share rows](./networking.md#availabledata) is parsed into the [actual data structures](./data_structures.md#availabledata); if parsing fails here, the block is invalid. TODO define fraud proof for invalid block here From 777cc696ae196a5cc5634c4fcac712e619ea9ec8 Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 18:52:34 -0500 Subject: [PATCH 13/51] Add rule for available data. --- specs/consensus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/consensus.md b/specs/consensus.md index ea25339..7eda109 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -139,7 +139,7 @@ TODO define fraud proof for invalid block here Once parsed, the following checks must be `true`: -1. +1. The commitments of the [erasure-coded extended](./data_structures.md#2d-reed-solomon-encoding-scheme) `availableData` must match those in `header.availableDataHeader`. ### State Transitions From 0adcc79ccc7ee9264c1d4cbbf33664b97010509d Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 20:15:55 -0500 Subject: [PATCH 14/51] Refactor available data original size field. --- specs/consensus.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 7eda109..aac5331 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -98,14 +98,15 @@ If the above fields are parsed successfully, the available data `block.available The [block header](./data_structures.md#header) `block.header` (`header` for short) is the first thing that is downloaded from the new block, and commits to everything inside the block in some way. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: +`availableDataOriginalSquareSize` is computed as described [here](./data_structures.md#header). + 1. `header.height` == `prev.header.height + 1`. 1. `header.timestamp` > `prev.header.timestamp`. 1. `header.lastBlockID` == the [block ID](./data_structures.md#blockid) of `prev`. 1. `header.lastCommitHash` == the [hash](./data_structures.md#hashing) of `lastCommit`. 1. `header.consensusRoot` == the value computed [here](./data_structures.md#consensus-parameters). 1. `header.stateCommitment` == the root of the state, computed [with the application of all state transitions in this block](#state-transitions). -1. `header.availableDataOriginalSquareSize` <= [`AVAILABLE_DATA_ORIGINAL_SQUARE_MAX`](#constants). -1. `header.availableDataOriginalSquareSize` is a power of `2`. +1. `availableDataOriginalSquareSize` <= [`AVAILABLE_DATA_ORIGINAL_SQUARE_MAX`](#constants). 1. `header.availableDataRoot` == TODO available data root 1. `header.proposerAddress` == the [leader](#leader-selection) for `header.height`. @@ -115,7 +116,7 @@ TODO define the genesis block The [available data header](./data_structures.md#availabledataheader)) `block.availableDataHeader` (`availableDataHeader` for short) is then processed. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. The following checks must be `true`: -1. Length of `availableDataHeader.availableDataCommitments` == `2 * header.availableDataOriginalSquareSize`. +1. Length of `availableDataHeader.availableDataCommitments` == `2 * availableDataOriginalSquareSize`. 1. The length of each element in `availableDataHeader.availableDataCommitments` must be [`32`](./consensus.md#hashing). #### `block.lastCommit` From 5a757207b5beb50d39e6a722b4b78dc7baed68ee Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 20:19:07 -0500 Subject: [PATCH 15/51] Minor cleanup for data structures figure. --- specs/figures/block_data_structures.dot | 4 ++-- specs/figures/block_data_structures.svg | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/figures/block_data_structures.dot b/specs/figures/block_data_structures.dot index cc70b77..c94d945 100644 --- a/specs/figures/block_data_structures.dot +++ b/specs/figures/block_data_structures.dot @@ -22,12 +22,12 @@ digraph G { subgraph cluster_header { label = "header"; - struct1 [label = "height | timestamp | lastBlockID | lastCommitHash | consensusRoot | stateCommitment | availableDataOriginalSharesUsed | availableDataRoot | proposerAddress"]; + struct1 [label = "height | timestamp | lastBlockID | lastCommitHash | consensusRoot | stateCommitment | availableDataOriginalSharesUsed | availableDataRoot | proposerAddress"]; } } struct1:f3 -> struct2; - struct1:f6 -> struct4 [label = "Merkle root of"]; + struct1:f7 -> struct4 [label = "Merkle root of"]; struct4:f0 -> struct3 [label = "NMT roots to\nerasure-coded data"]; edge [style = invis]; diff --git a/specs/figures/block_data_structures.svg b/specs/figures/block_data_structures.svg index 712d9bd..3e110df 100644 --- a/specs/figures/block_data_structures.svg +++ b/specs/figures/block_data_structures.svg @@ -96,7 +96,7 @@ -struct1:f6->struct4 +struct1:f7->struct4 Merkle root of From b566ec0ed0928bec159ea675a908307a0665690a Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 20:21:52 -0500 Subject: [PATCH 16/51] Add rule for lengths available data lists. --- specs/consensus.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specs/consensus.md b/specs/consensus.md index aac5331..e33638e 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -141,6 +141,7 @@ TODO define fraud proof for invalid block here Once parsed, the following checks must be `true`: 1. The commitments of the [erasure-coded extended](./data_structures.md#2d-reed-solomon-encoding-scheme) `availableData` must match those in `header.availableDataHeader`. +1. Length of `availableData.intermediateStateRootData` == length of `availableData.transactionData` + length of `availableData.evidenceData`. ### State Transitions From e2d3866e28619e97919ccff88cee883618e76a5b Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 20:25:09 -0500 Subject: [PATCH 17/51] Clean up. --- specs/consensus.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index e33638e..2cd3619 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -16,9 +16,9 @@ Consensus Rules - [`block.lastCommit`](#blocklastcommit) - [`block.availableData`](#blockavailabledata) - [State Transitions](#state-transitions) + - [`block.availableData.evidenceData`](#blockavailabledataevidencedata) + - [`block.availableData.transactionData`](#blockavailabledatatransactiondata) - [Validators and Delegations](#validators-and-delegations) - - [Formatting](#formatting) - - [Availability](#availability) ## System Parameters @@ -140,11 +140,19 @@ TODO define fraud proof for invalid block here Once parsed, the following checks must be `true`: -1. The commitments of the [erasure-coded extended](./data_structures.md#2d-reed-solomon-encoding-scheme) `availableData` must match those in `header.availableDataHeader`. +1. The commitments of the [erasure-coded extended](./data_structures.md#2d-reed-solomon-encoding-scheme) `availableData` must match those in `header.availableDataHeader`. Implicitly, this means that both rows and columns must be ordered lexicographically by namespace ID since they are committed to in a [Namespace Merkle Tree](data_structures.md#namespace-merkle-tree). 1. Length of `availableData.intermediateStateRootData` == length of `availableData.transactionData` + length of `availableData.evidenceData`. ### State Transitions +#### `block.availableData.evidenceData` + + + +#### `block.availableData.transactionData` + + + #### Validators and Delegations A transaction `tx` that requests a new validator initializes a new [Validator](data_structures.md#validator) leaf in the inactive validators subtree for that account as follows: @@ -217,9 +225,3 @@ then updates the target validator's voting power: validator.delegatedCount -= 1 validator.votingPower -= delegation.votingPower ``` - -### Formatting - -Leaves in the message [Namespace Merkle Tree](data_structures.md#namespace-merkle-tree) must be ordered lexicographically by namespace ID. - -### Availability From 6ef80838a3cf0e53ca53a0911e96b79e5b8932e9 Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 20:25:42 -0500 Subject: [PATCH 18/51] Clean up. --- specs/consensus.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 2cd3619..bb92e77 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -48,7 +48,6 @@ Consensus Rules | `UNBONDING_DURATION` | `uint32` | | `block` | Duration, in blocks, for unbonding a validator or delegation. | | `VERSION` | `uint64` | `1` | | Version of the LazyLedger chain. Breaking changes (hard forks) must update this parameter. | - ### Reserved Namespace IDs | name | type | value | description | @@ -67,7 +66,6 @@ Consensus Rules | `ACTIVE_VALIDATORS_SUBTREE_ID` | `StateSubtreeID` | `0x02` | | `INACTIVE_VALIDATORS_SUBTREE_ID` | `StateSubtreeID` | `0x03` | - ### Rewards and Penalties | name | type | value | unit | description | @@ -77,8 +75,12 @@ Consensus Rules ## Leader Selection +TODO + ## Fork Choice +TODO + ## Block Validity This section specifies the entirety of the validity rules for a newly-seen block, `block`. From 4d49b4a65528091baadf6c7fa1a1c4247822157d Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 20:37:54 -0500 Subject: [PATCH 19/51] Add minor introduction for state transitions. --- specs/consensus.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/specs/consensus.md b/specs/consensus.md index bb92e77..400d644 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -147,12 +147,18 @@ Once parsed, the following checks must be `true`: ### State Transitions +Once the basic structure of the block [has been validated](#block-structure), state transitions must be applied to compute the new state and state root. + #### `block.availableData.evidenceData` +Evidence is the first set of state transitions that are applied, and represent proof of validator misbehavior. +TODO process evidence #### `block.availableData.transactionData` +Once [evidence has been processed](#blockavailabledataevidencedata), transactions are applied to the state. Note that _transactions_ mutate the state (essentially, the validator set and minimal balances), while _messages_ do not. See [the architecture documentation](./architecture.md) for more info. + #### Validators and Delegations From de21ed0c86f4ec1d993047d6f219358e838a5e2d Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 10 Nov 2020 20:43:29 -0500 Subject: [PATCH 20/51] Clarify how available data is parsed. --- specs/consensus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/consensus.md b/specs/consensus.md index 400d644..3dc328a 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -136,7 +136,7 @@ TODO define specifically how to validate commitsigs #### `block.availableData` -The block's [available data](./data_structures.md#availabledata) (analogous to transactions in contemporary blockchain designs) `block.availableData` (`availableData` for short) is finally processed. The [list of share rows](./networking.md#availabledata) is parsed into the [actual data structures](./data_structures.md#availabledata); if parsing fails here, the block is invalid. +The block's [available data](./data_structures.md#availabledata) (analogous to transactions in contemporary blockchain designs) `block.availableData` (`availableData` for short) is finally processed. The [list of share rows](./networking.md#availabledata) is parsed into the [actual data structures](./data_structures.md#availabledata) using the reverse of [the process to encode available data into shares](./data_structures.md#arranging-available-data-into-shares); if parsing fails here, the block is invalid. TODO define fraud proof for invalid block here From 3e28226f9c7bc6b096ba7d2db3f41244c2db47dd Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 11 Nov 2020 10:56:26 -0500 Subject: [PATCH 21/51] Fix available data header rows and cols roots. --- specs/consensus.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 3dc328a..b59fb89 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -118,8 +118,9 @@ TODO define the genesis block The [available data header](./data_structures.md#availabledataheader)) `block.availableDataHeader` (`availableDataHeader` for short) is then processed. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. The following checks must be `true`: -1. Length of `availableDataHeader.availableDataCommitments` == `2 * availableDataOriginalSquareSize`. -1. The length of each element in `availableDataHeader.availableDataCommitments` must be [`32`](./consensus.md#hashing). +1. Length of `availableDataHeader.rowRoots` == `ceil(availableDataOriginalSharesUsed / availableDataOriginalSquareSize) + availableDataOriginalSquareSize`. +1. Length of `availableDataHeader.colRoots` == `availableDataOriginalSquareSize * 2`. +1. The length of each element in `availableDataHeader.rowRoots` and `availableDataHeader.colRoots` must be [`32`](./consensus.md#hashing). #### `block.lastCommit` @@ -159,8 +160,6 @@ TODO process evidence Once [evidence has been processed](#blockavailabledataevidencedata), transactions are applied to the state. Note that _transactions_ mutate the state (essentially, the validator set and minimal balances), while _messages_ do not. See [the architecture documentation](./architecture.md) for more info. - - #### Validators and Delegations A transaction `tx` that requests a new validator initializes a new [Validator](data_structures.md#validator) leaf in the inactive validators subtree for that account as follows: From 355553ce3e48b45cb77c2631e7e149e4a38f21c9 Mon Sep 17 00:00:00 2001 From: John Adler Date: Sun, 15 Nov 2020 09:50:13 -0500 Subject: [PATCH 22/51] Add rules for basic validation of transaction data. --- specs/consensus.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/specs/consensus.md b/specs/consensus.md index b59fb89..5cc6b96 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -160,6 +160,16 @@ TODO process evidence Once [evidence has been processed](#blockavailabledataevidencedata), transactions are applied to the state. Note that _transactions_ mutate the state (essentially, the validator set and minimal balances), while _messages_ do not. See [the architecture documentation](./architecture.md) for more info. +`block.availableData.transactionData` is simply a list of [WrappedTransaction](./data_structures.md#wrappedtransaction)s. For each wrapped transaction in this list, `wrappedTransaction`, with index `i` (starting from `0`), the following checks must be `true`: + +1. `wrappedTransaction.index` == `i`. + +For `wrappedTransaction`'s [transaction](./data_structures.md#transaction) `transaction`, the following checks must be `true`: + +1. `transaction.signature` must be a [valid signature](./data_structures.md#public-key-cryptography) over `transaction.signedTransactionData`. + +TODO add some logic for signing over implicit data, e.g. chain ID + #### Validators and Delegations A transaction `tx` that requests a new validator initializes a new [Validator](data_structures.md#validator) leaf in the inactive validators subtree for that account as follows: From f3e1864b8bcb46e3e6c293e6aba45957488b8d7d Mon Sep 17 00:00:00 2001 From: John Adler Date: Sun, 15 Nov 2020 09:55:17 -0500 Subject: [PATCH 23/51] Reorganize headers. --- specs/consensus.md | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 5cc6b96..359a46c 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -10,15 +10,15 @@ Consensus Rules - [Leader Selection](#leader-selection) - [Fork Choice](#fork-choice) - [Block Validity](#block-validity) - - [Block Structure](#block-structure) - - [`block.header`](#blockheader) - - [`block.availableDataHeader`](#blockavailabledataheader) - - [`block.lastCommit`](#blocklastcommit) - - [`block.availableData`](#blockavailabledata) - - [State Transitions](#state-transitions) - - [`block.availableData.evidenceData`](#blockavailabledataevidencedata) - - [`block.availableData.transactionData`](#blockavailabledatatransactiondata) - - [Validators and Delegations](#validators-and-delegations) +- [Block Structure](#block-structure) + - [`block.header`](#blockheader) + - [`block.availableDataHeader`](#blockavailabledataheader) + - [`block.lastCommit`](#blocklastcommit) + - [`block.availableData`](#blockavailabledata) +- [State Transitions](#state-transitions) + - [`block.availableData.evidenceData`](#blockavailabledataevidencedata) + - [`block.availableData.transactionData`](#blockavailabledatatransactiondata) + - [Validators and Delegations](#validators-and-delegations) ## System Parameters @@ -83,9 +83,11 @@ TODO ## Block Validity -This section specifies the entirety of the validity rules for a newly-seen block, `block`. +The validity of a newly-seen block, `block`, is determined by two components, detailed in subsequent sections: +1. [Block structure](#block-structure): whether the block header is valid, and data in a block is arranged into a valid and matching data root (i.e. syntax). +1. [State transition](#state-transitions): whether the application of transactions in the block produces a matching and valid state root (i.e. semantics). -### Block Structure +## Block Structure Before executing [state transitions](#state-transitions), the structure of the [block](./data_structures.md#block) must be verified. @@ -96,7 +98,7 @@ The following block fields are acquired from the network and parsed (i.e. [deser If the above fields are parsed successfully, the available data `block.availableData` is acquired in erasure-coded form as [a list of share rows](./networking.md#availabledata), then parsed. If it cannot be parsed, the block is ignored but not explicitly invalid, as above. -#### `block.header` +### `block.header` The [block header](./data_structures.md#header) `block.header` (`header` for short) is the first thing that is downloaded from the new block, and commits to everything inside the block in some way. For previous block `prev` (if `prev` is not known, then the block is ignored), and previous block header `prev.header`, the following checks must be `true`: @@ -114,7 +116,7 @@ The [block header](./data_structures.md#header) `block.header` (`header` for sho TODO define the genesis block -#### `block.availableDataHeader` +### `block.availableDataHeader` The [available data header](./data_structures.md#availabledataheader)) `block.availableDataHeader` (`availableDataHeader` for short) is then processed. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. The following checks must be `true`: @@ -122,7 +124,7 @@ The [available data header](./data_structures.md#availabledataheader)) `block.av 1. Length of `availableDataHeader.colRoots` == `availableDataOriginalSquareSize * 2`. 1. The length of each element in `availableDataHeader.rowRoots` and `availableDataHeader.colRoots` must be [`32`](./consensus.md#hashing). -#### `block.lastCommit` +### `block.lastCommit` The last [commit](./data_structures.md#commit) `block.lastCommit` (`lastCommit` for short) is processed next. This is the Tendermint commit (i.e. polka of votes) _for the previous block_. For previous block `prev` and previous block header `prev.header`, the following checks must be `true`: @@ -135,7 +137,7 @@ The last [commit](./data_structures.md#commit) `block.lastCommit` (`lastCommit` TODO define specifically how to validate commitsigs -#### `block.availableData` +### `block.availableData` The block's [available data](./data_structures.md#availabledata) (analogous to transactions in contemporary blockchain designs) `block.availableData` (`availableData` for short) is finally processed. The [list of share rows](./networking.md#availabledata) is parsed into the [actual data structures](./data_structures.md#availabledata) using the reverse of [the process to encode available data into shares](./data_structures.md#arranging-available-data-into-shares); if parsing fails here, the block is invalid. @@ -146,17 +148,17 @@ Once parsed, the following checks must be `true`: 1. The commitments of the [erasure-coded extended](./data_structures.md#2d-reed-solomon-encoding-scheme) `availableData` must match those in `header.availableDataHeader`. Implicitly, this means that both rows and columns must be ordered lexicographically by namespace ID since they are committed to in a [Namespace Merkle Tree](data_structures.md#namespace-merkle-tree). 1. Length of `availableData.intermediateStateRootData` == length of `availableData.transactionData` + length of `availableData.evidenceData`. -### State Transitions +## State Transitions Once the basic structure of the block [has been validated](#block-structure), state transitions must be applied to compute the new state and state root. -#### `block.availableData.evidenceData` +### `block.availableData.evidenceData` Evidence is the first set of state transitions that are applied, and represent proof of validator misbehavior. TODO process evidence -#### `block.availableData.transactionData` +### `block.availableData.transactionData` Once [evidence has been processed](#blockavailabledataevidencedata), transactions are applied to the state. Note that _transactions_ mutate the state (essentially, the validator set and minimal balances), while _messages_ do not. See [the architecture documentation](./architecture.md) for more info. @@ -170,7 +172,7 @@ For `wrappedTransaction`'s [transaction](./data_structures.md#transaction) `tran TODO add some logic for signing over implicit data, e.g. chain ID -#### Validators and Delegations +### Validators and Delegations A transaction `tx` that requests a new validator initializes a new [Validator](data_structures.md#validator) leaf in the inactive validators subtree for that account as follows: ``` From 8e018792bd39244c115f2320666407502e0cc1c5 Mon Sep 17 00:00:00 2001 From: John Adler Date: Sun, 15 Nov 2020 10:22:09 -0500 Subject: [PATCH 24/51] Add skeleton for processing transactions. --- specs/consensus.md | 52 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/specs/consensus.md b/specs/consensus.md index 359a46c..3774e08 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -18,6 +18,16 @@ Consensus Rules - [State Transitions](#state-transitions) - [`block.availableData.evidenceData`](#blockavailabledataevidencedata) - [`block.availableData.transactionData`](#blockavailabledatatransactiondata) + - [SignedTransactionDataTransfer](#signedtransactiondatatransfer) + - [SignedTransactionDataPayForMessage](#signedtransactiondatapayformessage) + - [SignedTransactionDataPayForPadding](#signedtransactiondatapayforpadding) + - [SignedTransactionDataCreateValidator](#signedtransactiondatacreatevalidator) + - [SignedTransactionDataBeginUnbondingValidator](#signedtransactiondatabeginunbondingvalidator) + - [SignedTransactionDataUnbondValidator](#signedtransactiondataunbondvalidator) + - [SignedTransactionDataCreateDelegation](#signedtransactiondatacreatedelegation) + - [SignedTransactionDataBeginUnbondingDelegation](#signedtransactiondatabeginunbondingdelegation) + - [SignedTransactionDataUnbondDelegation](#signedtransactiondataunbonddelegation) + - [SignedTransactionDataBurn](#signedtransactiondataburn) - [Validators and Delegations](#validators-and-delegations) ## System Parameters @@ -172,6 +182,48 @@ For `wrappedTransaction`'s [transaction](./data_structures.md#transaction) `tran TODO add some logic for signing over implicit data, e.g. chain ID +Finally, each `wrappedTransaction` is processed depending on [its type](./data_structures.mdsignedtransactiondata). These are specified in the next subsections. + +#### SignedTransactionDataTransfer + +TODO + +#### SignedTransactionDataPayForMessage + +TODO + +#### SignedTransactionDataPayForPadding + +TODO + +#### SignedTransactionDataCreateValidator + +TODO + +#### SignedTransactionDataBeginUnbondingValidator + +TODO + +#### SignedTransactionDataUnbondValidator + +TODO + +#### SignedTransactionDataCreateDelegation + +TODO + +#### SignedTransactionDataBeginUnbondingDelegation + +TODO + +#### SignedTransactionDataUnbondDelegation + +TODO + +#### SignedTransactionDataBurn + +TODO + ### Validators and Delegations A transaction `tx` that requests a new validator initializes a new [Validator](data_structures.md#validator) leaf in the inactive validators subtree for that account as follows: From 49fa3b3ea6b206059272af18f1a3a5518e6d4b23 Mon Sep 17 00:00:00 2001 From: John Adler Date: Sun, 15 Nov 2020 10:22:51 -0500 Subject: [PATCH 25/51] Minor wording fix. --- specs/consensus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/consensus.md b/specs/consensus.md index 3774e08..783b286 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -182,7 +182,7 @@ For `wrappedTransaction`'s [transaction](./data_structures.md#transaction) `tran TODO add some logic for signing over implicit data, e.g. chain ID -Finally, each `wrappedTransaction` is processed depending on [its type](./data_structures.mdsignedtransactiondata). These are specified in the next subsections. +Finally, each `wrappedTransaction` is processed depending on [its transaction type](./data_structures.mdsignedtransactiondata). These are specified in the next subsections. #### SignedTransactionDataTransfer From 974b5fb9f6dec0f5a8102e0f487e9e08565cd41c Mon Sep 17 00:00:00 2001 From: John Adler Date: Mon, 16 Nov 2020 12:13:57 -0500 Subject: [PATCH 26/51] Fix bad link. --- specs/consensus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/consensus.md b/specs/consensus.md index 783b286..629f3bb 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -182,7 +182,7 @@ For `wrappedTransaction`'s [transaction](./data_structures.md#transaction) `tran TODO add some logic for signing over implicit data, e.g. chain ID -Finally, each `wrappedTransaction` is processed depending on [its transaction type](./data_structures.mdsignedtransactiondata). These are specified in the next subsections. +Finally, each `wrappedTransaction` is processed depending on [its transaction type](./data_structures.md#signedtransactiondata). These are specified in the next subsections. #### SignedTransactionDataTransfer From 4f4971b32f312a3506e09ef72c7b12f2fd83657f Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 17 Nov 2020 15:55:21 -0500 Subject: [PATCH 27/51] Add state access shorthand for use with state transitions. --- specs/consensus.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/specs/consensus.md b/specs/consensus.md index 629f3bb..4e501f8 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -162,6 +162,8 @@ Once parsed, the following checks must be `true`: Once the basic structure of the block [has been validated](#block-structure), state transitions must be applied to compute the new state and state root. +For this section, the variable `state` represents the [state tree](./data_structures.md#state), with `state.accounts[k]`, `state.accounts[k]`, and `state.accounts[k]` being shorthand for the leaf in the state tree in the [accounts, inactive validator set, and active validator set subtrees](./data_structures.md#state) with key `k`. E.g. `state.accounts[a]` is shorthand for `state[(ACCOUNTS_SUBTREE_ID << 8*(32-STATE_SUBTREE_RESERVED_BYTES)) | ((-1 >> 8*STATE_SUBTREE_RESERVED_BYTES) & a)]`. + ### `block.availableData.evidenceData` Evidence is the first set of state transitions that are applied, and represent proof of validator misbehavior. @@ -184,6 +186,8 @@ TODO add some logic for signing over implicit data, e.g. chain ID Finally, each `wrappedTransaction` is processed depending on [its transaction type](./data_structures.md#signedtransactiondata). These are specified in the next subsections. +TODO handle fees + #### SignedTransactionDataTransfer TODO From c90a5d9cdef7eee5c9a6623f3c8c451b171ae122 Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 17 Nov 2020 21:58:07 -0500 Subject: [PATCH 28/51] Add some partial rules for txs. --- specs/consensus.md | 127 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 114 insertions(+), 13 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 4e501f8..07f9ef5 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -162,7 +162,7 @@ Once parsed, the following checks must be `true`: Once the basic structure of the block [has been validated](#block-structure), state transitions must be applied to compute the new state and state root. -For this section, the variable `state` represents the [state tree](./data_structures.md#state), with `state.accounts[k]`, `state.accounts[k]`, and `state.accounts[k]` being shorthand for the leaf in the state tree in the [accounts, inactive validator set, and active validator set subtrees](./data_structures.md#state) with key `k`. E.g. `state.accounts[a]` is shorthand for `state[(ACCOUNTS_SUBTREE_ID << 8*(32-STATE_SUBTREE_RESERVED_BYTES)) | ((-1 >> 8*STATE_SUBTREE_RESERVED_BYTES) & a)]`. +For this section, the variable `state` represents the [state tree](./data_structures.md#state), with `state.accounts[k]`, `state.accounts[k]`, and `state.accounts[k]` being shorthand for the leaf in the state tree in the [accounts, inactive validator set, and active validator set subtrees](./data_structures.md#state) with [pre-hashed key](./data_structures.md#state) `k`. E.g. `state.accounts[a]` is shorthand for `state[(ACCOUNTS_SUBTREE_ID << 8*(32-STATE_SUBTREE_RESERVED_BYTES)) | ((-1 >> 8*STATE_SUBTREE_RESERVED_BYTES) & hash(a))]`. ### `block.availableData.evidenceData` @@ -184,49 +184,150 @@ For `wrappedTransaction`'s [transaction](./data_structures.md#transaction) `tran TODO add some logic for signing over implicit data, e.g. chain ID -Finally, each `wrappedTransaction` is processed depending on [its transaction type](./data_structures.md#signedtransactiondata). These are specified in the next subsections. +Finally, each `wrappedTransaction` is processed depending on [its transaction type](./data_structures.md#signedtransactiondata). These are specified in the next subsections, where `tx` is short for `transaction.signedTransactionData`, and `sender` is the recovered signing [address](./data_structures.md#address). After applying a transaction, the new state state root is computed. -TODO handle fees +TODO **handle fees** + +TODO logic to handle intermediate state roots #### SignedTransactionDataTransfer -TODO +The following checks must be `true`: + +1. `tx.type` == [`TransactionType.Transfer`](./data_structures.md#signedtransactiondata). +1. `tx.amount` <= `state.accounts[sender].balance`. +1. `tx.nonce` == `state.accounts[sender].nonce + 1`. + +Apply the following to the state: + +1. `state.accounts[sender].balance -= tx.amount`. +1. `state.accounts[sender].nonce += 1`. +1. `state.accounts[tx.to].balance += tx.amount`. #### SignedTransactionDataPayForMessage -TODO +The following checks must be `true`: + +1. `tx.type` == [`TransactionType.PayForMessage`](./data_structures.md#signedtransactiondata). +1. `tx.nonce` == `state.accounts[sender].nonce + 1`. +1. The `ceil(tx.messageSize / SHARE_SIZE)` shares starting at index `wrappedTransactions.messageStartIndex` must: + 1. Have have namespace ID `tx.messageNamespaceID`. +1. `tx.messageShareCommitment` == computed as described [here](./data_structures.md#signedtransactiondatapayformessage). + +Apply the following to the state: + +1. `state.accounts[sender].nonce += 1`. #### SignedTransactionDataPayForPadding -TODO +The following checks must be `true`: + +1. `tx.type` == [`TransactionType.PayForPadding`](./data_structures.md#signedtransactiondata). +1. `tx.` +1. `tx.` +1. `tx.` +1. `tx.` + +Apply the following to the state: + +1. `state.accounts[sender].nonce += 1`. #### SignedTransactionDataCreateValidator -TODO +The following checks must be `true`: + +1. `tx.type` == [`TransactionType.CreateValidator`](./data_structures.md#signedtransactiondata). +1. `tx.` +1. `tx.` +1. `tx.` +1. `tx.` + +Apply the following to the state: + +1. `state.accounts[sender].nonce += 1`. #### SignedTransactionDataBeginUnbondingValidator -TODO +The following checks must be `true`: + +1. `tx.type` == [`TransactionType.BeginUnbondingValidator`](./data_structures.md#signedtransactiondata). +1. `tx.` +1. `tx.` +1. `tx.` +1. `tx.` + +Apply the following to the state: + +1. `state.accounts[sender].nonce += 1`. #### SignedTransactionDataUnbondValidator -TODO +The following checks must be `true`: + +1. `tx.type` == [`TransactionType.UnbondValidator`](./data_structures.md#signedtransactiondata). +1. `tx.` +1. `tx.` +1. `tx.` +1. `tx.` + +Apply the following to the state: + +1. `state.accounts[sender].nonce += 1`. #### SignedTransactionDataCreateDelegation -TODO +The following checks must be `true`: + +1. `tx.type` == [`TransactionType.CreateDelegation`](./data_structures.md#signedtransactiondata). +1. `tx.` +1. `tx.` +1. `tx.` +1. `tx.` + +Apply the following to the state: + +1. `state.accounts[sender].nonce += 1`. #### SignedTransactionDataBeginUnbondingDelegation -TODO +The following checks must be `true`: + +1. `tx.type` == [`TransactionType.BeginUnbondingDelegation`](./data_structures.md#signedtransactiondata). +1. `tx.` +1. `tx.` +1. `tx.` +1. `tx.` + +Apply the following to the state: + +1. `state.accounts[sender].nonce += 1`. #### SignedTransactionDataUnbondDelegation -TODO +The following checks must be `true`: + +1. `tx.type` == [`TransactionType.UnbondDelegation`](./data_structures.md#signedtransactiondata). +1. `tx.` +1. `tx.` +1. `tx.` +1. `tx.` + +Apply the following to the state: + +1. `state.accounts[sender].nonce += 1`. #### SignedTransactionDataBurn -TODO +The following checks must be `true`: + +1. `tx.type` == [`TransactionType.Burn`](./data_structures.md#signedtransactiondata). +1. `tx.amount` <= `state.accounts[sender].balance`. +1. `tx.nonce` == `state.accounts[sender].nonce + 1`. + +Apply the following to the state: + +1. `state.accounts[sender].balance -= tx.amount`. +1. `state.accounts[sender].nonce += 1`. ### Validators and Delegations From d0b9b862f57aa261d4665e50bfb569f3a50890d4 Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 17 Nov 2020 22:00:34 -0500 Subject: [PATCH 29/51] Remove pay for padding tx. --- specs/consensus.md | 15 --------------- specs/data_structures.md | 27 +++++++-------------------- 2 files changed, 7 insertions(+), 35 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 07f9ef5..c95220e 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -20,7 +20,6 @@ Consensus Rules - [`block.availableData.transactionData`](#blockavailabledatatransactiondata) - [SignedTransactionDataTransfer](#signedtransactiondatatransfer) - [SignedTransactionDataPayForMessage](#signedtransactiondatapayformessage) - - [SignedTransactionDataPayForPadding](#signedtransactiondatapayforpadding) - [SignedTransactionDataCreateValidator](#signedtransactiondatacreatevalidator) - [SignedTransactionDataBeginUnbondingValidator](#signedtransactiondatabeginunbondingvalidator) - [SignedTransactionDataUnbondValidator](#signedtransactiondataunbondvalidator) @@ -218,20 +217,6 @@ Apply the following to the state: 1. `state.accounts[sender].nonce += 1`. -#### SignedTransactionDataPayForPadding - -The following checks must be `true`: - -1. `tx.type` == [`TransactionType.PayForPadding`](./data_structures.md#signedtransactiondata). -1. `tx.` -1. `tx.` -1. `tx.` -1. `tx.` - -Apply the following to the state: - -1. `state.accounts[sender].nonce += 1`. - #### SignedTransactionDataCreateValidator The following checks must be `true`: diff --git a/specs/data_structures.md b/specs/data_structures.md index afe5362..cde2313 100644 --- a/specs/data_structures.md +++ b/specs/data_structures.md @@ -40,7 +40,6 @@ Data Structures - [SignedTransactionData](#signedtransactiondata) - [SignedTransactionDataTransfer](#signedtransactiondatatransfer) - [SignedTransactionDataPayForMessage](#signedtransactiondatapayformessage) - - [SignedTransactionDataPayForPadding](#signedtransactiondatapayforpadding) - [SignedTransactionDataCreateValidator](#signedtransactiondatacreatevalidator) - [SignedTransactionDataBeginUnbondingValidator](#signedtransactiondatabeginunbondingvalidator) - [SignedTransactionDataUnbondValidator](#signedtransactiondataunbondvalidator) @@ -540,21 +539,19 @@ Wrapped transactions include additional metadata by the block proposer that is c enum TransactionType : uint8_t { Transfer = 1, PayForMessage = 2, - PayForPadding = 3, - CreateValidator = 4, - BeginUnbondingValidator = 5, - UnbondValidator = 6, - CreateDelegation = 7, - BeginUnbondingDelegation = 8, - UnbondDelegation = 9, - Burn = 10, + CreateValidator = 3, + BeginUnbondingValidator = 4, + UnbondValidator = 5, + CreateDelegation = 6, + BeginUnbondingDelegation = 7, + UnbondDelegation = 8, + Burn = 9, }; ``` Signed transaction data comes in a number of types: 1. [Transfer](#signedtransactiondatatransfer) 1. [PayForMessage](#signedtransactiondatapayformessage) -1. [PayForPadding](#signedtransactiondatapayforpadding) 1. [CreateValidator](#signedtransactiondatacreatevalidator) 1. [BeginUnbondingValidator](#signedtransactiondatabeginunbondingvalidator) 1. [UnbondValidator](#signedtransactiondataunbondvalidator) @@ -600,16 +597,6 @@ Pays for the inclusion of a [message](#message) in the same block. The commitment to message shares `messageShareCommitment` is a [Merkle root](#binary-merkle-tree) of message share roots. Each message share root is [a subtree root in a row NMT](#arranging-available-data-into-shares). For rationale, see [rationale doc](../rationale/message_block_layout.md). -##### SignedTransactionDataPayForPadding - -| name | type | description | -| -------------------- | ------------------------------ | ------------------------------------------------------------ | -| `type` | `TransactionType` | Must be `TransactionType.PayForPadding`. | -| `messageNamespaceID` | [`NamespaceID`](#type-aliases) | Namespace ID of padding this transaction pays a fee for. | -| `messageSize` | `uint64` | Size of padding this transaction pays a fee for, in `byte`s. | - -Pays for the inclusion of a padding shares in the same block. Padding shares are used between real messages that are not tightly packed. For rationale, see [rationale doc](../rationale/message_block_layout.md). - ##### SignedTransactionDataCreateValidator | name | type | description | From 343a3f76ea8a9036418d936d946bbcd758774f87 Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 17 Nov 2020 22:08:20 -0500 Subject: [PATCH 30/51] Make padding shares deterministic. --- rationale/message_block_layout.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/rationale/message_block_layout.md b/rationale/message_block_layout.md index a6cd3b9..d264829 100644 --- a/rationale/message_block_layout.md +++ b/rationale/message_block_layout.md @@ -46,4 +46,8 @@ The last piece of the puzzle is determining _which_ row the message is placed at ### Caveats -The message placement rules described above conflict with the first rule that shares must be ordered by namespace ID, as shares between two messages that are not placed adjacent to each other do not have a natural namespace they belong to. This is resolved by having special transactions the block producer includes that specify a range a zero-padding shares for a given namespace ID (which must be between the namespace IDs of the surrounding real shares, inclusive). +The message placement rules described above conflict with the first rule that shares must be ordered by namespace ID, as shares between two messages that are not placed adjacent to each other do not have a natural namespace they belong to. This is resolved with two rules: +1. the namespace ID for message shares must be "even" (i.e. the lowest bit must be `0`), and +1. the namespace ID for shares between messages must be exactly `1` more than the namespace ID of the preceding message (i.e. must be identical save for the lowest bit being `1` instead of `0`). + +If we interpret namespace IDs as unsigned integers, this can be thought of as restricting the namespace IDs of shares belonging to messages to even namespace IDs and padding shares to odd namespace IDs, further restricted to deterministic values. From 68303b9a0895907088ac027012ac1b09b02a095e Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 17 Nov 2020 22:23:07 -0500 Subject: [PATCH 31/51] Extend rule to handle padding between txs and messages. --- rationale/message_block_layout.md | 5 +++-- specs/consensus.md | 17 +++++++++-------- specs/data_structures.md | 2 +- 3 files changed, 13 insertions(+), 11 deletions(-) diff --git a/rationale/message_block_layout.md b/rationale/message_block_layout.md index d264829..1dbb5c9 100644 --- a/rationale/message_block_layout.md +++ b/rationale/message_block_layout.md @@ -46,8 +46,9 @@ The last piece of the puzzle is determining _which_ row the message is placed at ### Caveats -The message placement rules described above conflict with the first rule that shares must be ordered by namespace ID, as shares between two messages that are not placed adjacent to each other do not have a natural namespace they belong to. This is resolved with two rules: +The message placement rules described above conflict with the first rule that shares must be ordered by namespace ID, as shares between two messages that are not placed adjacent to each other do not have a natural namespace they belong to. This is resolved with some rules: 1. the namespace ID for message shares must be "even" (i.e. the lowest bit must be `0`), and -1. the namespace ID for shares between messages must be exactly `1` more than the namespace ID of the preceding message (i.e. must be identical save for the lowest bit being `1` instead of `0`). +1. the namespace ID for shares between messages must be exactly `1` more than the namespace ID of the preceding message (i.e. must be identical save for the lowest bit being `1` instead of `0`), and. +1. the namespace ID for shares between requests with a reserved namespace ID and messages must be [`TAIL_TRANSACTION_PADDING_NAMESPACE_ID`](../specs/consensus.md#constants). Note that this only applies for shares _between_ requests with a reserved namespace ID and messages, i.e. if there are no messages in the block then this rule does not apply. If we interpret namespace IDs as unsigned integers, this can be thought of as restricting the namespace IDs of shares belonging to messages to even namespace IDs and padding shares to odd namespace IDs, further restricted to deterministic values. diff --git a/specs/consensus.md b/specs/consensus.md index c95220e..443dc4a 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -50,7 +50,7 @@ Consensus Rules | `GRAFFITI_BYTES` | `uint64` | `32` | `byte` | Maximum size of transaction graffiti, in bytes. | | `MAX_VALIDATORS` | `uint16` | `64` | | Maximum number of active validators. | | `NAMESPACE_ID_BYTES` | `uint64` | `8` | `byte` | Size of namespace ID, in bytes. | -| `NAMESPACE_ID_MAX_RESERVED` | `uint64` | `255` | | Value of maximum reserved namespace ID (inclusive). 1 byte worth of IDs. | +| `NAMESPACE_ID_MAX_RESERVED` | `uint64` | `254` | | Value of maximum reserved namespace ID (inclusive). 1 byte worth of IDs. | | `SHARE_RESERVED_BYTES` | `uint64` | `1` | `byte` | Bytes reserved at the beginning of each [share](data_structures.md#share). Must be sufficient to represent `SHARE_SIZE`. | | `SHARE_SIZE` | `uint64` | `256` | `byte` | Size of transaction and message [shares](data_structures.md#share), in bytes. | | `STATE_SUBTREE_RESERVED_BYTES` | `uint64` | `1` | `byte` | Number of bytes reserved to identify state subtrees. | @@ -59,13 +59,14 @@ Consensus Rules ### Reserved Namespace IDs -| name | type | value | description | -| -------------------------------------- | ------------- | -------------------- | ----------------------------------------------------------------------------- | -| `TRANSACTION_NAMESPACE_ID` | `NamespaceID` | `0x0000000000000001` | Transactions: requests that modify the state. | -| `INTERMEDIATE_STATE_ROOT_NAMESPACE_ID` | `NamespaceID` | `0x0000000000000002` | Intermediate state roots, committed after every transaction. | -| `EVIDENCE_NAMESPACE_ID` | `NamespaceID` | `0x0000000000000003` | Evidence: fraud proofs or other proof of slashable action. | -| `TAIL_PADDING_NAMESPACE_ID` | `NamespaceID` | `0xFFFFFFFFFFFFFFFE` | Tail padding: padding after all messages to fill up the original data square. | -| `PARITY_SHARE_NAMESPACE_ID` | `NamespaceID` | `0xFFFFFFFFFFFFFFFF` | Parity shares: extended shares in the available data matrix. | +| name | type | value | description | +| --------------------------------------- | ------------- | -------------------- | ------------------------------------------------------------------------------------------ | +| `TRANSACTION_NAMESPACE_ID` | `NamespaceID` | `0x0000000000000001` | Transactions: requests that modify the state. | +| `INTERMEDIATE_STATE_ROOT_NAMESPACE_ID` | `NamespaceID` | `0x0000000000000002` | Intermediate state roots, committed after every transaction. | +| `EVIDENCE_NAMESPACE_ID` | `NamespaceID` | `0x0000000000000003` | Evidence: fraud proofs or other proof of slashable action. | +| `TAIL_TRANSACTION_PADDING_NAMESPACE_ID` | `NamespaceID` | `0x00000000000000FF` | Tail padding for transactions: padding after all transactions but before messages. | +| `TAIL_PADDING_NAMESPACE_ID` | `NamespaceID` | `0xFFFFFFFFFFFFFFFE` | Tail padding for messages: padding after all messages to fill up the original data square. | +| `PARITY_SHARE_NAMESPACE_ID` | `NamespaceID` | `0xFFFFFFFFFFFFFFFF` | Parity shares: extended shares in the available data matrix. | ### Reserved State Subtree IDs diff --git a/specs/data_structures.md b/specs/data_structures.md index cde2313..8f7042d 100644 --- a/specs/data_structures.md +++ b/specs/data_structures.md @@ -506,7 +506,7 @@ In the example below, two messages (of lengths 2 and 1, respectively) are placed ![fig: Original data: messages.](./figures/rs2d_originaldata_message.svg) -The non-interactive default rules may introduce empty shares that do not belong to any message (in the example above, the top-right share is empty). These must be explicitly disclaimed by the block producer using [special transactions](#signedtransactiondata-payforpadding). +The non-interactive default rules may introduce empty shares that do not belong to any message (in the example above, the top-right share is empty). These are zeroes with namespace ID equal to the either . See the [rationale doc](../rationale/message_block_layout.md) for more info. ## Available Data From 8154a609361e161acf19e00477109d3939ade02d Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 17 Nov 2020 22:24:37 -0500 Subject: [PATCH 32/51] Clarify rules around padding in data structures doc. --- specs/data_structures.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/data_structures.md b/specs/data_structures.md index 8f7042d..3ab4541 100644 --- a/specs/data_structures.md +++ b/specs/data_structures.md @@ -506,7 +506,7 @@ In the example below, two messages (of lengths 2 and 1, respectively) are placed ![fig: Original data: messages.](./figures/rs2d_originaldata_message.svg) -The non-interactive default rules may introduce empty shares that do not belong to any message (in the example above, the top-right share is empty). These are zeroes with namespace ID equal to the either . See the [rationale doc](../rationale/message_block_layout.md) for more info. +The non-interactive default rules may introduce empty shares that do not belong to any message (in the example above, the top-right share is empty). These are zeroes with namespace ID equal to the either [`TAIL_TRANSACTION_PADDING_NAMESPACE_ID`](./consensus.md#constants) if between a request with a reserved namespace ID and a message, or the namespace ID of the previous message plus `1`. See the [rationale doc](../rationale/message_block_layout.md) for more info. ## Available Data From bf76dbc52f249348d824a6e6fe059a291645f0a3 Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 17 Nov 2020 22:26:20 -0500 Subject: [PATCH 33/51] Add consensus check for tx message namespace ID. --- specs/consensus.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specs/consensus.md b/specs/consensus.md index 443dc4a..8c06824 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -210,6 +210,7 @@ The following checks must be `true`: 1. `tx.type` == [`TransactionType.PayForMessage`](./data_structures.md#signedtransactiondata). 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. +1. `uint(tx.messageNamespaceID) % 2` == `0`. In other words, the namespace ID interpreted as an unsigned integer is even. 1. The `ceil(tx.messageSize / SHARE_SIZE)` shares starting at index `wrappedTransactions.messageStartIndex` must: 1. Have have namespace ID `tx.messageNamespaceID`. 1. `tx.messageShareCommitment` == computed as described [here](./data_structures.md#signedtransactiondatapayformessage). From b33ce2b241d0eab9bd0cb1bbee03f551ed2de31c Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 17 Nov 2020 22:27:51 -0500 Subject: [PATCH 34/51] Fix typo. --- specs/consensus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/consensus.md b/specs/consensus.md index 8c06824..7b0c78a 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -212,7 +212,7 @@ The following checks must be `true`: 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. 1. `uint(tx.messageNamespaceID) % 2` == `0`. In other words, the namespace ID interpreted as an unsigned integer is even. 1. The `ceil(tx.messageSize / SHARE_SIZE)` shares starting at index `wrappedTransactions.messageStartIndex` must: - 1. Have have namespace ID `tx.messageNamespaceID`. + 1. Have namespace ID `tx.messageNamespaceID`. 1. `tx.messageShareCommitment` == computed as described [here](./data_structures.md#signedtransactiondatapayformessage). Apply the following to the state: From 9073da8f5e5d2fe11191d325fa78ee972350cbbe Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 17 Nov 2020 22:35:09 -0500 Subject: [PATCH 35/51] Add rule for tx create validator. --- specs/consensus.md | 44 +++++++++++++++++++++----------------------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 7b0c78a..6e77436 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -162,7 +162,7 @@ Once parsed, the following checks must be `true`: Once the basic structure of the block [has been validated](#block-structure), state transitions must be applied to compute the new state and state root. -For this section, the variable `state` represents the [state tree](./data_structures.md#state), with `state.accounts[k]`, `state.accounts[k]`, and `state.accounts[k]` being shorthand for the leaf in the state tree in the [accounts, inactive validator set, and active validator set subtrees](./data_structures.md#state) with [pre-hashed key](./data_structures.md#state) `k`. E.g. `state.accounts[a]` is shorthand for `state[(ACCOUNTS_SUBTREE_ID << 8*(32-STATE_SUBTREE_RESERVED_BYTES)) | ((-1 >> 8*STATE_SUBTREE_RESERVED_BYTES) & hash(a))]`. +For this section, the variable `state` represents the [state tree](./data_structures.md#state), with `state.accounts[k]`, `state.inactiveValidatorSet[k]`, and `state.activeValidatorSet[k]` being shorthand for the leaf in the state tree in the [accounts, inactive validator set, and active validator set subtrees](./data_structures.md#state) with [pre-hashed key](./data_structures.md#state) `k`. E.g. `state.accounts[a]` is shorthand for `state[(ACCOUNTS_SUBTREE_ID << 8*(32-STATE_SUBTREE_RESERVED_BYTES)) | ((-1 >> 8*STATE_SUBTREE_RESERVED_BYTES) & hash(a))]`. ### `block.availableData.evidenceData` @@ -224,21 +224,32 @@ Apply the following to the state: The following checks must be `true`: 1. `tx.type` == [`TransactionType.CreateValidator`](./data_structures.md#signedtransactiondata). -1. `tx.` -1. `tx.` -1. `tx.` -1. `tx.` +1. `tx.amount` <= `state.accounts[sender].balance`. +1. `tx.nonce` == `state.accounts[sender].nonce + 1`. +1. `tx.commissionRate` TODO check some bounds here Apply the following to the state: 1. `state.accounts[sender].nonce += 1`. +1. Initialize a new [Validator](data_structures.md#validator) leaf in the inactive validators subtree for the sender account `state.inactiveValidatorSet[sender]` (`validator` for short): +``` +validator.status = ValidatorStatus.Queued +validator.stakedBalance = tx.amount +validator.commissionRate = tx.commissionRate +validator.delegatedCount = 0 +validator.votingPower = tx.amount +validator.pendingRewards = 0 +validator.latestEntry = PeriodEntry(0) +validator.unbondingHeight = 0 +validator.isSlashed = false +``` #### SignedTransactionDataBeginUnbondingValidator The following checks must be `true`: 1. `tx.type` == [`TransactionType.BeginUnbondingValidator`](./data_structures.md#signedtransactiondata). -1. `tx.` +1. `tx.nonce` == `state.accounts[sender].nonce + 1`. 1. `tx.` 1. `tx.` 1. `tx.` @@ -252,7 +263,7 @@ Apply the following to the state: The following checks must be `true`: 1. `tx.type` == [`TransactionType.UnbondValidator`](./data_structures.md#signedtransactiondata). -1. `tx.` +1. `tx.nonce` == `state.accounts[sender].nonce + 1`. 1. `tx.` 1. `tx.` 1. `tx.` @@ -266,7 +277,7 @@ Apply the following to the state: The following checks must be `true`: 1. `tx.type` == [`TransactionType.CreateDelegation`](./data_structures.md#signedtransactiondata). -1. `tx.` +1. `tx.nonce` == `state.accounts[sender].nonce + 1`. 1. `tx.` 1. `tx.` 1. `tx.` @@ -280,7 +291,7 @@ Apply the following to the state: The following checks must be `true`: 1. `tx.type` == [`TransactionType.BeginUnbondingDelegation`](./data_structures.md#signedtransactiondata). -1. `tx.` +1. `tx.nonce` == `state.accounts[sender].nonce + 1`. 1. `tx.` 1. `tx.` 1. `tx.` @@ -294,7 +305,7 @@ Apply the following to the state: The following checks must be `true`: 1. `tx.type` == [`TransactionType.UnbondDelegation`](./data_structures.md#signedtransactiondata). -1. `tx.` +1. `tx.nonce` == `state.accounts[sender].nonce + 1`. 1. `tx.` 1. `tx.` 1. `tx.` @@ -318,19 +329,6 @@ Apply the following to the state: ### Validators and Delegations -A transaction `tx` that requests a new validator initializes a new [Validator](data_structures.md#validator) leaf in the inactive validators subtree for that account as follows: -``` -validator.status = ValidatorStatus.Queued -validator.stakedBalance = tx.amount -validator.commissionRate = tx.commissionRate -validator.delegatedCount = 0 -validator.votingPower = tx.amount -validator.pendingRewards = 0 -validator.latestEntry = PeriodEntry(0) -validator.unbondingHeight = 0 -validator.isSlashed = false -``` - At the end of a block at the end of an epoch, the top `MAX_VALIDATORS` validators by voting power are or become active (bonded). For newly-bonded validators, the entire validator object is moved to the active validators subtree and their status is changed to bonded. ``` validator.status = ValidatorStatus.Bonded From d47e3f9d003a8d07928a3a5a8d33f02b425e4c8f Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 17 Nov 2020 23:10:31 -0500 Subject: [PATCH 36/51] Add todo for end block. --- specs/consensus.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/consensus.md b/specs/consensus.md index 6e77436..0f0d5c7 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -158,6 +158,8 @@ Once parsed, the following checks must be `true`: 1. The commitments of the [erasure-coded extended](./data_structures.md#2d-reed-solomon-encoding-scheme) `availableData` must match those in `header.availableDataHeader`. Implicitly, this means that both rows and columns must be ordered lexicographically by namespace ID since they are committed to in a [Namespace Merkle Tree](data_structures.md#namespace-merkle-tree). 1. Length of `availableData.intermediateStateRootData` == length of `availableData.transactionData` + length of `availableData.evidenceData`. +TODO add a step for BLOCK_END state transition + ## State Transitions Once the basic structure of the block [has been validated](#block-structure), state transitions must be applied to compute the new state and state root. From 3b0a96ba88b43b8abb4391ff7dae84c504513a05 Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 18 Nov 2020 09:41:25 -0500 Subject: [PATCH 37/51] Add some rules for unbond tx. --- specs/consensus.md | 92 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 73 insertions(+), 19 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 0f0d5c7..64a58f9 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -27,6 +27,7 @@ Consensus Rules - [SignedTransactionDataBeginUnbondingDelegation](#signedtransactiondatabeginunbondingdelegation) - [SignedTransactionDataUnbondDelegation](#signedtransactiondataunbonddelegation) - [SignedTransactionDataBurn](#signedtransactiondataburn) + - [End Block](#end-block) - [Validators and Delegations](#validators-and-delegations) ## System Parameters @@ -202,9 +203,12 @@ The following checks must be `true`: Apply the following to the state: -1. `state.accounts[sender].balance -= tx.amount`. -1. `state.accounts[sender].nonce += 1`. -1. `state.accounts[tx.to].balance += tx.amount`. +``` +state.accounts[sender].nonce += 1 + +state.accounts[sender].balance -= tx.amount +state.accounts[tx.to].balance += tx.amount +``` #### SignedTransactionDataPayForMessage @@ -219,7 +223,9 @@ The following checks must be `true`: Apply the following to the state: -1. `state.accounts[sender].nonce += 1`. +``` +state.accounts[sender].nonce += 1 +``` #### SignedTransactionDataCreateValidator @@ -232,9 +238,11 @@ The following checks must be `true`: Apply the following to the state: -1. `state.accounts[sender].nonce += 1`. -1. Initialize a new [Validator](data_structures.md#validator) leaf in the inactive validators subtree for the sender account `state.inactiveValidatorSet[sender]` (`validator` for short): ``` +state.accounts[sender].nonce += 1 + +validator = new Validator + validator.status = ValidatorStatus.Queued validator.stakedBalance = tx.amount validator.commissionRate = tx.commissionRate @@ -244,6 +252,8 @@ validator.pendingRewards = 0 validator.latestEntry = PeriodEntry(0) validator.unbondingHeight = 0 validator.isSlashed = false + +state.inactiveValidatorSet[sender] = validator ``` #### SignedTransactionDataBeginUnbondingValidator @@ -252,13 +262,29 @@ The following checks must be `true`: 1. `tx.type` == [`TransactionType.BeginUnbondingValidator`](./data_structures.md#signedtransactiondata). 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. -1. `tx.` -1. `tx.` -1. `tx.` +1. `state.inactiveValidatorSet[sender]` == `ValidatorStatus.Queued` or `state.activeValidatorSet[sender]` == `ValidatorStatus.Bonded`. Apply the following to the state: -1. `state.accounts[sender].nonce += 1`. +``` +state.accounts[sender].nonce += 1 + +if state.inactiveValidatorSet[sender].status == ValidatorStatus.Queued + validator = state.inactiveValidatorSet[sender] +else if state.activeValidatorSet[sender].status == ValidatorStatus.Bonded + validator = state.activeValidatorSet[sender] + delete state.activeValidatorSet[sender] + +validator.status = ValidatorStatus.Unbonding +validator.unbondingHeight = block.height + 1 + +old_pendingRewards = validator.pendingRewards +old_votingPower = validator.votingPower +validator.pendingRewards = 0 +validator.latestEntry += old_pendingRewards // old_votingPower + +state.inactiveValidatorSet[sender] = validator +``` #### SignedTransactionDataUnbondValidator @@ -266,13 +292,25 @@ The following checks must be `true`: 1. `tx.type` == [`TransactionType.UnbondValidator`](./data_structures.md#signedtransactiondata). 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. -1. `tx.` -1. `tx.` -1. `tx.` +1. `state.inactiveValidatorSet[sender]` == `ValidatorStatus.Unbonding`. +1. `state.inactiveValidatorSet[sender].unbondingHeight + UNBONDING_DURATION > block.height`. Apply the following to the state: -1. `state.accounts[sender].nonce += 1`. +``` +state.accounts[sender].nonce += 1 + +validator = state.inactiveValidatorSet[sender] + +old_stakedBalance = validator.stakedBalance +validator.status = ValidatorStatus.Unbonded +validator.stakedBalance = 0 +validator.votingPower -= old_stakedBalance + +state.inactiveValidatorSet[sender] = validator + +state.accounts[sender].balance += old_stakedBalance +``` #### SignedTransactionDataCreateDelegation @@ -286,7 +324,9 @@ The following checks must be `true`: Apply the following to the state: -1. `state.accounts[sender].nonce += 1`. +``` +state.accounts[sender].nonce += 1 +``` #### SignedTransactionDataBeginUnbondingDelegation @@ -300,7 +340,9 @@ The following checks must be `true`: Apply the following to the state: -1. `state.accounts[sender].nonce += 1`. +``` +state.accounts[sender].nonce += 1 +``` #### SignedTransactionDataUnbondDelegation @@ -314,7 +356,9 @@ The following checks must be `true`: Apply the following to the state: -1. `state.accounts[sender].nonce += 1`. +``` +state.accounts[sender].nonce += 1 +``` #### SignedTransactionDataBurn @@ -326,8 +370,15 @@ The following checks must be `true`: Apply the following to the state: -1. `state.accounts[sender].balance -= tx.amount`. -1. `state.accounts[sender].nonce += 1`. +``` +state.accounts[sender].nonce += 1 + +state.accounts[sender].balance -= tx.amount +``` + +#### End Block + + ### Validators and Delegations @@ -360,6 +411,9 @@ validator.pendingRewards = 0 validator.latestEntry += old_pendingRewards / old_votingPower ``` + + + A transaction `tx` that requests a new delegation first updates the target validator's voting power: ``` validator.delegatedCount += 1 From 2d63069173e29173249a4304ba245c3cf01fa523 Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 18 Nov 2020 09:49:08 -0500 Subject: [PATCH 38/51] Rearrange. --- specs/consensus.md | 74 +++++++++++++++++++--------------------------- 1 file changed, 30 insertions(+), 44 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 64a58f9..07f0176 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -328,6 +328,22 @@ Apply the following to the state: state.accounts[sender].nonce += 1 ``` +A transaction `tx` that requests a new delegation first updates the target validator's voting power: +``` +validator.delegatedCount += 1 +validator.votingPower += tx.amount +``` + +then initializes the [Delegation](data_structures.md#delegation) field of that account as follows: +``` +delegation.status = DelegationStatus.Bonded +delegation.validator = tx.to +delegation.stakedBalance = tx.amount +delegation.beginEntry = validator.latestEntry +delegation.endEntry = PeriodEntry(0) +delegation.unbondingHeight = 0 +``` + #### SignedTransactionDataBeginUnbondingDelegation The following checks must be `true`: @@ -344,6 +360,19 @@ Apply the following to the state: state.accounts[sender].nonce += 1 ``` +A transaction `tx` that requests withdrawing a delegation first updates the delegation field: +``` +delegation.status = DelegationStatus.Unbonding +delegation.endEntry = validator.latestEntry +delegation.unbondingHeight = block.height + 1 +``` + +then updates the target validator's voting power: +``` +validator.delegatedCount -= 1 +validator.votingPower -= delegation.votingPower +``` + #### SignedTransactionDataUnbondDelegation The following checks must be `true`: @@ -378,10 +407,6 @@ state.accounts[sender].balance -= tx.amount #### End Block - - -### Validators and Delegations - At the end of a block at the end of an epoch, the top `MAX_VALIDATORS` validators by voting power are or become active (bonded). For newly-bonded validators, the entire validator object is moved to the active validators subtree and their status is changed to bonded. ``` validator.status = ValidatorStatus.Bonded @@ -393,14 +418,7 @@ validator.status = ValidatorStatus.Unbonding validator.unbondingHeight = block.height + 1 ``` -Once an unbonding validator has waited at least `UNBONDING_DURATION` blocks, they can be unbonded, collecting their reward: -``` -old_stakedBalance = validator.stakedBalance - -validator.status = ValidatorStatus.Unbonded -validator.stakedBalance = 0 -validator.votingPower -= old_stakedBalance -``` +### Validators and Delegations Every time a bonded validator's voting power changes (i.e. when a delegation is added or removed), or when a validator begins unbonding, the rewards per unit of voting power accumulated so far are calculated: ``` @@ -410,35 +428,3 @@ old_votingPower = validator.votingPower validator.pendingRewards = 0 validator.latestEntry += old_pendingRewards / old_votingPower ``` - - - - -A transaction `tx` that requests a new delegation first updates the target validator's voting power: -``` -validator.delegatedCount += 1 -validator.votingPower += tx.amount -``` - -then initializes the [Delegation](data_structures.md#delegation) field of that account as follows: -``` -delegation.status = DelegationStatus.Bonded -delegation.validator = tx.to -delegation.stakedBalance = tx.amount -delegation.beginEntry = validator.latestEntry -delegation.endEntry = PeriodEntry(0) -delegation.unbondingHeight = 0 -``` - -A transaction `tx` that requests withdrawing a delegation first updates the delegation field: -``` -delegation.status = DelegationStatus.Unbonding -delegation.endEntry = validator.latestEntry -delegation.unbondingHeight = block.height + 1 -``` - -then updates the target validator's voting power: -``` -validator.delegatedCount -= 1 -validator.votingPower -= delegation.votingPower -``` From 92b311c0af12f2e8a95cf9b5b57574b5c1d0f28f Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 18 Nov 2020 09:51:13 -0500 Subject: [PATCH 39/51] Add check for is validator when creating validator. --- specs/consensus.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/consensus.md b/specs/consensus.md index 07f0176..59b5999 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -235,11 +235,13 @@ The following checks must be `true`: 1. `tx.amount` <= `state.accounts[sender].balance`. 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. 1. `tx.commissionRate` TODO check some bounds here +1. `state.accounts[sender].isValidator` == `false` and `state.accounts[sender].isDelegating` == `false`. Apply the following to the state: ``` state.accounts[sender].nonce += 1 +state.accounts[sender].isValidator = true validator = new Validator From 8c4c90f3794e470d58f88d206fee48b7ef8cdb54 Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 18 Nov 2020 09:53:27 -0500 Subject: [PATCH 40/51] Logic to handle unbonding validator with no delegations. --- specs/consensus.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/specs/consensus.md b/specs/consensus.md index 59b5999..bc069af 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -312,6 +312,10 @@ validator.votingPower -= old_stakedBalance state.inactiveValidatorSet[sender] = validator state.accounts[sender].balance += old_stakedBalance + +if validator.delegatedCount == 0 + state.accounts[sender].isValidator = false + delete state.inactiveValidatorSet[sender] ``` #### SignedTransactionDataCreateDelegation From f9dba6e81c1e4bead9c58dae30824a6b1000a19a Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 18 Nov 2020 09:59:41 -0500 Subject: [PATCH 41/51] Add delegation creation. --- specs/consensus.md | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index bc069af..5642cd4 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -323,31 +323,41 @@ if validator.delegatedCount == 0 The following checks must be `true`: 1. `tx.type` == [`TransactionType.CreateDelegation`](./data_structures.md#signedtransactiondata). +1. `tx.amount` <= `state.accounts[sender].balance`. +1. `state.accounts[tx.to].isValidator == true`. +1. `state.inactiveValidatorSet[tx.to].status` == `ValidatorStatus.Queued` or `state.activeValidatorSet[tx.to].status` == `ValidatorStatus.Bonded` 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. -1. `tx.` -1. `tx.` -1. `tx.` +1. `state.accounts[sender].isValidator` == `false` and `state.accounts[sender].isDelegating` == `false`. Apply the following to the state: ``` state.accounts[sender].nonce += 1 -``` +state.accounts[sender].isDelegating = true + +if state.inactiveValidatorSet[tx.to].status == ValidatorStatus.Queued + validator = state.inactiveValidatorSet[tx.to] +else if state.activeValidatorSet[tx.to].status == ValidatorStatus.Bonded + validator = state.activeValidatorSet[tx.to] -A transaction `tx` that requests a new delegation first updates the target validator's voting power: -``` validator.delegatedCount += 1 validator.votingPower += tx.amount -``` -then initializes the [Delegation](data_structures.md#delegation) field of that account as follows: -``` +if state.inactiveValidatorSet[tx.to].status == ValidatorStatus.Queued + state.inactiveValidatorSet[tx.to] = validator +else if state.activeValidatorSet[tx.to].status == ValidatorStatus.Bonded + state.activeValidatorSet[tx.to] = validator + +delegation = new Delegation + delegation.status = DelegationStatus.Bonded delegation.validator = tx.to delegation.stakedBalance = tx.amount delegation.beginEntry = validator.latestEntry delegation.endEntry = PeriodEntry(0) delegation.unbondingHeight = 0 + +state.accounts[sender].delegationInfo = delegation ``` #### SignedTransactionDataBeginUnbondingDelegation From 403aab73f0ab3892a748c28eada2f232e2afde0b Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 18 Nov 2020 14:09:16 -0500 Subject: [PATCH 42/51] Add rules for delegations. Handle balance transfers for rest of txs. --- specs/consensus.md | 119 +++++++++++++++++++++++++-------------------- 1 file changed, 67 insertions(+), 52 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 5642cd4..1f930e4 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -28,7 +28,6 @@ Consensus Rules - [SignedTransactionDataUnbondDelegation](#signedtransactiondataunbonddelegation) - [SignedTransactionDataBurn](#signedtransactiondataburn) - [End Block](#end-block) - - [Validators and Delegations](#validators-and-delegations) ## System Parameters @@ -255,6 +254,8 @@ validator.latestEntry = PeriodEntry(0) validator.unbondingHeight = 0 validator.isSlashed = false +state.accounts[sender].balance -= tx.amount + state.inactiveValidatorSet[sender] = validator ``` @@ -279,11 +280,8 @@ else if state.activeValidatorSet[sender].status == ValidatorStatus.Bonded validator.status = ValidatorStatus.Unbonding validator.unbondingHeight = block.height + 1 - -old_pendingRewards = validator.pendingRewards -old_votingPower = validator.votingPower +validator.latestEntry += validator.pendingRewards // validator.votingPower validator.pendingRewards = 0 -validator.latestEntry += old_pendingRewards // old_votingPower state.inactiveValidatorSet[sender] = validator ``` @@ -295,7 +293,7 @@ The following checks must be `true`: 1. `tx.type` == [`TransactionType.UnbondValidator`](./data_structures.md#signedtransactiondata). 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. 1. `state.inactiveValidatorSet[sender]` == `ValidatorStatus.Unbonding`. -1. `state.inactiveValidatorSet[sender].unbondingHeight + UNBONDING_DURATION > block.height`. +1. `state.inactiveValidatorSet[sender].unbondingHeight + UNBONDING_DURATION < block.height`. Apply the following to the state: @@ -304,15 +302,14 @@ state.accounts[sender].nonce += 1 validator = state.inactiveValidatorSet[sender] -old_stakedBalance = validator.stakedBalance +state.accounts[sender].balance += validator.stakedBalance + validator.status = ValidatorStatus.Unbonded +validator.votingPower -= validator.stakedBalance validator.stakedBalance = 0 -validator.votingPower -= old_stakedBalance state.inactiveValidatorSet[sender] = validator -state.accounts[sender].balance += old_stakedBalance - if validator.delegatedCount == 0 state.accounts[sender].isValidator = false delete state.inactiveValidatorSet[sender] @@ -335,20 +332,14 @@ Apply the following to the state: state.accounts[sender].nonce += 1 state.accounts[sender].isDelegating = true -if state.inactiveValidatorSet[tx.to].status == ValidatorStatus.Queued - validator = state.inactiveValidatorSet[tx.to] -else if state.activeValidatorSet[tx.to].status == ValidatorStatus.Bonded - validator = state.activeValidatorSet[tx.to] +state.accounts[sender].balance -= tx.amount -validator.delegatedCount += 1 -validator.votingPower += tx.amount +delegation = new Delegation if state.inactiveValidatorSet[tx.to].status == ValidatorStatus.Queued - state.inactiveValidatorSet[tx.to] = validator + validator = state.inactiveValidatorSet[tx.to] else if state.activeValidatorSet[tx.to].status == ValidatorStatus.Bonded - state.activeValidatorSet[tx.to] = validator - -delegation = new Delegation + validator = state.activeValidatorSet[tx.to] delegation.status = DelegationStatus.Bonded delegation.validator = tx.to @@ -357,7 +348,17 @@ delegation.beginEntry = validator.latestEntry delegation.endEntry = PeriodEntry(0) delegation.unbondingHeight = 0 +validator.latestEntry += validator.pendingRewards // validator.votingPower +validator.pendingRewards = 0 +validator.delegatedCount += 1 +validator.votingPower += tx.amount + state.accounts[sender].delegationInfo = delegation + +if state.inactiveValidatorSet[tx.to].status == ValidatorStatus.Queued + state.inactiveValidatorSet[tx.to] = validator +else if state.activeValidatorSet[tx.to].status == ValidatorStatus.Bonded + state.activeValidatorSet[tx.to] = validator ``` #### SignedTransactionDataBeginUnbondingDelegation @@ -366,27 +367,40 @@ The following checks must be `true`: 1. `tx.type` == [`TransactionType.BeginUnbondingDelegation`](./data_structures.md#signedtransactiondata). 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. -1. `tx.` -1. `tx.` -1. `tx.` +1. `state.accounts[sender].isDelegating == true`. +1. `state.accounts[sender].delegationInfo.status` == `DelegationStatus.Bonded`. Apply the following to the state: ``` state.accounts[sender].nonce += 1 -``` -A transaction `tx` that requests withdrawing a delegation first updates the delegation field: -``` +delegation = state.accounts[sender].delegationInfo + +if state.inactiveValidatorSet[delegation.validator].status == ValidatorStatus.Queued || + state.inactiveValidatorSet[delegation.validator].status == ValidatorStatus.Unbonding + state.inactiveValidatorSet[delegation.validator].status == ValidatorStatus.Unbonded + validator = state.inactiveValidatorSet[delegation.validator] +else if state.activeValidatorSet[delegation.validator].status == ValidatorStatus.Bonded + validator = state.activeValidatorSet[delegation.validator] + delegation.status = DelegationStatus.Unbonding delegation.endEntry = validator.latestEntry delegation.unbondingHeight = block.height + 1 -``` -then updates the target validator's voting power: -``` +validator.latestEntry += validator.pendingRewards // validator.votingPower +validator.pendingRewards = 0 validator.delegatedCount -= 1 validator.votingPower -= delegation.votingPower + +state.accounts[sender].delegationInfo = delegation + +if state.inactiveValidatorSet[delegation.validator].status == ValidatorStatus.Queued || + state.inactiveValidatorSet[delegation.validator].status == ValidatorStatus.Unbonding + state.inactiveValidatorSet[delegation.validator].status == ValidatorStatus.Unbonded + state.inactiveValidatorSet[delegation.validator] = validator +else if state.activeValidatorSet[delegation.validator].status == ValidatorStatus.Bonded + state.activeValidatorSet[delegation.validator] = validator ``` #### SignedTransactionDataUnbondDelegation @@ -395,14 +409,33 @@ The following checks must be `true`: 1. `tx.type` == [`TransactionType.UnbondDelegation`](./data_structures.md#signedtransactiondata). 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. -1. `tx.` -1. `tx.` -1. `tx.` +1. `state.accounts[sender].isDelegating` == `true`. +1. `state.accounts[sender].delegationInfo.status` == `DelegationStatus.Unbonding`. +1. `state.accounts[sender].delegationInfo.unbondingHeight + UNBONDING_DURATION < block.height`. Apply the following to the state: ``` state.accounts[sender].nonce += 1 +state.accounts[sender].isDelegating = false + +delegation = state.accounts[sender].delegationInfo + +state.accounts[sender].balance += delegation.stakedBalance + +if state.inactiveValidatorSet[delegation.validator].status == ValidatorStatus.Queued || + state.inactiveValidatorSet[delegation.validator].status == ValidatorStatus.Unbonding + state.inactiveValidatorSet[delegation.validator].status == ValidatorStatus.Unbonded + validator = state.inactiveValidatorSet[delegation.validator] +else if state.activeValidatorSet[delegation.validator].status == ValidatorStatus.Bonded + validator = state.activeValidatorSet[delegation.validator] + +if validator.delegatedCount == 0 + if validator.status == ValidatorStatus.Unbonded + state.accounts[delegation.validator].isValidator = false + delete state.inactiveValidatorSet[delegation.validator] + +delete state.accounts[sender].delegationInfo ``` #### SignedTransactionDataBurn @@ -423,24 +456,6 @@ state.accounts[sender].balance -= tx.amount #### End Block -At the end of a block at the end of an epoch, the top `MAX_VALIDATORS` validators by voting power are or become active (bonded). For newly-bonded validators, the entire validator object is moved to the active validators subtree and their status is changed to bonded. -``` -validator.status = ValidatorStatus.Bonded -``` - -For validators that were bonded but are no longer (either by being outside the top `MAX_VALIDATORS` validators, through a transaction that requests unbonding, or by being slashing), the validator object is moved to the inactive validators subtree and they begin unbonding. -``` -validator.status = ValidatorStatus.Unbonding -validator.unbondingHeight = block.height + 1 -``` - -### Validators and Delegations +TODO end block -Every time a bonded validator's voting power changes (i.e. when a delegation is added or removed), or when a validator begins unbonding, the rewards per unit of voting power accumulated so far are calculated: -``` -old_pendingRewards = validator.pendingRewards -old_votingPower = validator.votingPower - -validator.pendingRewards = 0 -validator.latestEntry += old_pendingRewards / old_votingPower -``` +At the end of a block, the top `MAX_VALIDATORS` validators by voting power are or become active (bonded). For newly-bonded validators, the entire validator object is moved to the active validators subtree and their status is changed to bonded. From 07a85dbbb7cf2b732d39bab29a5f1db433c3bf0b Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 18 Nov 2020 18:23:21 -0500 Subject: [PATCH 43/51] Clean up todos. --- specs/consensus.md | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 1f930e4..a6f59d0 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -121,11 +121,9 @@ The [block header](./data_structures.md#header) `block.header` (`header` for sho 1. `header.consensusRoot` == the value computed [here](./data_structures.md#consensus-parameters). 1. `header.stateCommitment` == the root of the state, computed [with the application of all state transitions in this block](#state-transitions). 1. `availableDataOriginalSquareSize` <= [`AVAILABLE_DATA_ORIGINAL_SQUARE_MAX`](#constants). -1. `header.availableDataRoot` == TODO available data root +1. `header.availableDataRoot` == the [Merkle root](./data_structures.md#binary-merkle-tree) of the tree with the row and column roots of `block.availableDataHeader` as leaves. 1. `header.proposerAddress` == the [leader](#leader-selection) for `header.height`. -TODO define the genesis block - ### `block.availableDataHeader` The [available data header](./data_structures.md#availabledataheader)) `block.availableDataHeader` (`availableDataHeader` for short) is then processed. This commits to the available data, which is only downloaded after the [consensus commit](#blocklastcommit) is processed. The following checks must be `true`: @@ -145,20 +143,20 @@ The last [commit](./data_structures.md#commit) `block.lastCommit` (`lastCommit` 1. Each of `lastCommit.signatures` must be a valid [CommitSig](./data_structures.md#commitsig) 1. The sum of the votes for `prev` in `lastCommit` must be at least 2/3 (rounded up) of the voting power of `prev`'s next validator set. -TODO define specifically how to validate commitsigs + ### `block.availableData` The block's [available data](./data_structures.md#availabledata) (analogous to transactions in contemporary blockchain designs) `block.availableData` (`availableData` for short) is finally processed. The [list of share rows](./networking.md#availabledata) is parsed into the [actual data structures](./data_structures.md#availabledata) using the reverse of [the process to encode available data into shares](./data_structures.md#arranging-available-data-into-shares); if parsing fails here, the block is invalid. -TODO define fraud proof for invalid block here + Once parsed, the following checks must be `true`: 1. The commitments of the [erasure-coded extended](./data_structures.md#2d-reed-solomon-encoding-scheme) `availableData` must match those in `header.availableDataHeader`. Implicitly, this means that both rows and columns must be ordered lexicographically by namespace ID since they are committed to in a [Namespace Merkle Tree](data_structures.md#namespace-merkle-tree). 1. Length of `availableData.intermediateStateRootData` == length of `availableData.transactionData` + length of `availableData.evidenceData`. -TODO add a step for BLOCK_END state transition + ## State Transitions @@ -170,7 +168,7 @@ For this section, the variable `state` represents the [state tree](./data_struct Evidence is the first set of state transitions that are applied, and represent proof of validator misbehavior. -TODO process evidence + ### `block.availableData.transactionData` @@ -184,13 +182,13 @@ For `wrappedTransaction`'s [transaction](./data_structures.md#transaction) `tran 1. `transaction.signature` must be a [valid signature](./data_structures.md#public-key-cryptography) over `transaction.signedTransactionData`. -TODO add some logic for signing over implicit data, e.g. chain ID + Finally, each `wrappedTransaction` is processed depending on [its transaction type](./data_structures.md#signedtransactiondata). These are specified in the next subsections, where `tx` is short for `transaction.signedTransactionData`, and `sender` is the recovered signing [address](./data_structures.md#address). After applying a transaction, the new state state root is computed. -TODO **handle fees** + -TODO logic to handle intermediate state roots + #### SignedTransactionDataTransfer @@ -233,7 +231,7 @@ The following checks must be `true`: 1. `tx.type` == [`TransactionType.CreateValidator`](./data_structures.md#signedtransactiondata). 1. `tx.amount` <= `state.accounts[sender].balance`. 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. -1. `tx.commissionRate` TODO check some bounds here +1. `tx.commissionRate` 1. `state.accounts[sender].isValidator` == `false` and `state.accounts[sender].isDelegating` == `false`. Apply the following to the state: @@ -456,6 +454,4 @@ state.accounts[sender].balance -= tx.amount #### End Block -TODO end block - At the end of a block, the top `MAX_VALIDATORS` validators by voting power are or become active (bonded). For newly-bonded validators, the entire validator object is moved to the active validators subtree and their status is changed to bonded. From 1e2df94ca0a26d4a38aa9ca52ac4642dee4691a1 Mon Sep 17 00:00:00 2001 From: John Adler Date: Wed, 18 Nov 2020 18:56:43 -0500 Subject: [PATCH 44/51] Remove some todos based on issues filed. --- specs/consensus.md | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index a6f59d0..77f220a 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -143,21 +143,15 @@ The last [commit](./data_structures.md#commit) `block.lastCommit` (`lastCommit` 1. Each of `lastCommit.signatures` must be a valid [CommitSig](./data_structures.md#commitsig) 1. The sum of the votes for `prev` in `lastCommit` must be at least 2/3 (rounded up) of the voting power of `prev`'s next validator set. - - ### `block.availableData` The block's [available data](./data_structures.md#availabledata) (analogous to transactions in contemporary blockchain designs) `block.availableData` (`availableData` for short) is finally processed. The [list of share rows](./networking.md#availabledata) is parsed into the [actual data structures](./data_structures.md#availabledata) using the reverse of [the process to encode available data into shares](./data_structures.md#arranging-available-data-into-shares); if parsing fails here, the block is invalid. - - Once parsed, the following checks must be `true`: 1. The commitments of the [erasure-coded extended](./data_structures.md#2d-reed-solomon-encoding-scheme) `availableData` must match those in `header.availableDataHeader`. Implicitly, this means that both rows and columns must be ordered lexicographically by namespace ID since they are committed to in a [Namespace Merkle Tree](data_structures.md#namespace-merkle-tree). 1. Length of `availableData.intermediateStateRootData` == length of `availableData.transactionData` + length of `availableData.evidenceData`. - - ## State Transitions Once the basic structure of the block [has been validated](#block-structure), state transitions must be applied to compute the new state and state root. @@ -168,8 +162,6 @@ For this section, the variable `state` represents the [state tree](./data_struct Evidence is the first set of state transitions that are applied, and represent proof of validator misbehavior. - - ### `block.availableData.transactionData` Once [evidence has been processed](#blockavailabledataevidencedata), transactions are applied to the state. Note that _transactions_ mutate the state (essentially, the validator set and minimal balances), while _messages_ do not. See [the architecture documentation](./architecture.md) for more info. @@ -182,14 +174,8 @@ For `wrappedTransaction`'s [transaction](./data_structures.md#transaction) `tran 1. `transaction.signature` must be a [valid signature](./data_structures.md#public-key-cryptography) over `transaction.signedTransactionData`. - - Finally, each `wrappedTransaction` is processed depending on [its transaction type](./data_structures.md#signedtransactiondata). These are specified in the next subsections, where `tx` is short for `transaction.signedTransactionData`, and `sender` is the recovered signing [address](./data_structures.md#address). After applying a transaction, the new state state root is computed. - - - - #### SignedTransactionDataTransfer The following checks must be `true`: From 37d31ca7d1839b5deb7ec8610f3cfdeb737f03d1 Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 24 Nov 2020 13:36:42 -0500 Subject: [PATCH 45/51] Clean up wording around evidence. --- specs/consensus.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/consensus.md b/specs/consensus.md index 77f220a..6286b56 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -160,7 +160,8 @@ For this section, the variable `state` represents the [state tree](./data_struct ### `block.availableData.evidenceData` -Evidence is the first set of state transitions that are applied, and represent proof of validator misbehavior. +Evidence is the first set of state transitions that are applied, ahead of [transactions](#blockavailabledatatransactiondata). +Each evidence represents proof of validator misbehavior, and causes a penalty against the validator(s). ### `block.availableData.transactionData` From 0ebc93f60783fee5b390f51f66098637d6c61efa Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 24 Nov 2020 13:37:21 -0500 Subject: [PATCH 46/51] Fix indentation. --- specs/consensus.md | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 6286b56..b569f1c 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -2,32 +2,32 @@ Consensus Rules === - [System Parameters](#system-parameters) - - [Units](#units) - - [Constants](#constants) - - [Reserved Namespace IDs](#reserved-namespace-ids) - - [Reserved State Subtree IDs](#reserved-state-subtree-ids) - - [Rewards and Penalties](#rewards-and-penalties) + - [Units](#units) + - [Constants](#constants) + - [Reserved Namespace IDs](#reserved-namespace-ids) + - [Reserved State Subtree IDs](#reserved-state-subtree-ids) + - [Rewards and Penalties](#rewards-and-penalties) - [Leader Selection](#leader-selection) - [Fork Choice](#fork-choice) - [Block Validity](#block-validity) - [Block Structure](#block-structure) - - [`block.header`](#blockheader) - - [`block.availableDataHeader`](#blockavailabledataheader) - - [`block.lastCommit`](#blocklastcommit) - - [`block.availableData`](#blockavailabledata) + - [`block.header`](#blockheader) + - [`block.availableDataHeader`](#blockavailabledataheader) + - [`block.lastCommit`](#blocklastcommit) + - [`block.availableData`](#blockavailabledata) - [State Transitions](#state-transitions) - - [`block.availableData.evidenceData`](#blockavailabledataevidencedata) - - [`block.availableData.transactionData`](#blockavailabledatatransactiondata) - - [SignedTransactionDataTransfer](#signedtransactiondatatransfer) - - [SignedTransactionDataPayForMessage](#signedtransactiondatapayformessage) - - [SignedTransactionDataCreateValidator](#signedtransactiondatacreatevalidator) - - [SignedTransactionDataBeginUnbondingValidator](#signedtransactiondatabeginunbondingvalidator) - - [SignedTransactionDataUnbondValidator](#signedtransactiondataunbondvalidator) - - [SignedTransactionDataCreateDelegation](#signedtransactiondatacreatedelegation) - - [SignedTransactionDataBeginUnbondingDelegation](#signedtransactiondatabeginunbondingdelegation) - - [SignedTransactionDataUnbondDelegation](#signedtransactiondataunbonddelegation) - - [SignedTransactionDataBurn](#signedtransactiondataburn) - - [End Block](#end-block) + - [`block.availableData.evidenceData`](#blockavailabledataevidencedata) + - [`block.availableData.transactionData`](#blockavailabledatatransactiondata) + - [SignedTransactionDataTransfer](#signedtransactiondatatransfer) + - [SignedTransactionDataPayForMessage](#signedtransactiondatapayformessage) + - [SignedTransactionDataCreateValidator](#signedtransactiondatacreatevalidator) + - [SignedTransactionDataBeginUnbondingValidator](#signedtransactiondatabeginunbondingvalidator) + - [SignedTransactionDataUnbondValidator](#signedtransactiondataunbondvalidator) + - [SignedTransactionDataCreateDelegation](#signedtransactiondatacreatedelegation) + - [SignedTransactionDataBeginUnbondingDelegation](#signedtransactiondatabeginunbondingdelegation) + - [SignedTransactionDataUnbondDelegation](#signedtransactiondataunbonddelegation) + - [SignedTransactionDataBurn](#signedtransactiondataburn) + - [End Block](#end-block) ## System Parameters From 6d9648a454bcbfef79172152a4f129253cb7962d Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 24 Nov 2020 13:44:49 -0500 Subject: [PATCH 47/51] Revert changes to inter-message padding shares. --- rationale/message_block_layout.md | 7 +------ specs/data_structures.md | 27 ++++++++++++++++++++------- 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/rationale/message_block_layout.md b/rationale/message_block_layout.md index 1dbb5c9..79da12b 100644 --- a/rationale/message_block_layout.md +++ b/rationale/message_block_layout.md @@ -46,9 +46,4 @@ The last piece of the puzzle is determining _which_ row the message is placed at ### Caveats -The message placement rules described above conflict with the first rule that shares must be ordered by namespace ID, as shares between two messages that are not placed adjacent to each other do not have a natural namespace they belong to. This is resolved with some rules: -1. the namespace ID for message shares must be "even" (i.e. the lowest bit must be `0`), and -1. the namespace ID for shares between messages must be exactly `1` more than the namespace ID of the preceding message (i.e. must be identical save for the lowest bit being `1` instead of `0`), and. -1. the namespace ID for shares between requests with a reserved namespace ID and messages must be [`TAIL_TRANSACTION_PADDING_NAMESPACE_ID`](../specs/consensus.md#constants). Note that this only applies for shares _between_ requests with a reserved namespace ID and messages, i.e. if there are no messages in the block then this rule does not apply. - -If we interpret namespace IDs as unsigned integers, this can be thought of as restricting the namespace IDs of shares belonging to messages to even namespace IDs and padding shares to odd namespace IDs, further restricted to deterministic values. +The message placement rules described above conflict with the first rule that shares must be ordered by namespace ID, as shares between two messages that are not placed adjacent to each other do not have a natural namespace they belong to. This is resolved by having [special transactions](./../specs/data_structures.md#signedtransactiondatapayforpadding) the block producer includes that specify a range a zero-padding shares for a given namespace ID (which must be between the namespace IDs of the surrounding real shares, inclusive). diff --git a/specs/data_structures.md b/specs/data_structures.md index 3ab4541..ee758ad 100644 --- a/specs/data_structures.md +++ b/specs/data_structures.md @@ -40,6 +40,7 @@ Data Structures - [SignedTransactionData](#signedtransactiondata) - [SignedTransactionDataTransfer](#signedtransactiondatatransfer) - [SignedTransactionDataPayForMessage](#signedtransactiondatapayformessage) + - [SignedTransactionDataPayForPadding](#signedtransactiondatapayforpadding) - [SignedTransactionDataCreateValidator](#signedtransactiondatacreatevalidator) - [SignedTransactionDataBeginUnbondingValidator](#signedtransactiondatabeginunbondingvalidator) - [SignedTransactionDataUnbondValidator](#signedtransactiondataunbondvalidator) @@ -539,19 +540,21 @@ Wrapped transactions include additional metadata by the block proposer that is c enum TransactionType : uint8_t { Transfer = 1, PayForMessage = 2, - CreateValidator = 3, - BeginUnbondingValidator = 4, - UnbondValidator = 5, - CreateDelegation = 6, - BeginUnbondingDelegation = 7, - UnbondDelegation = 8, - Burn = 9, + PayForPadding = 3, + CreateValidator = 4, + BeginUnbondingValidator = 5, + UnbondValidator = 6, + CreateDelegation = 7, + BeginUnbondingDelegation = 8, + UnbondDelegation = 9, + Burn = 10, }; ``` Signed transaction data comes in a number of types: 1. [Transfer](#signedtransactiondatatransfer) 1. [PayForMessage](#signedtransactiondatapayformessage) +1. [PayForPadding](#signedtransactiondatapayforpadding) 1. [CreateValidator](#signedtransactiondatacreatevalidator) 1. [BeginUnbondingValidator](#signedtransactiondatabeginunbondingvalidator) 1. [UnbondValidator](#signedtransactiondataunbondvalidator) @@ -597,6 +600,16 @@ Pays for the inclusion of a [message](#message) in the same block. The commitment to message shares `messageShareCommitment` is a [Merkle root](#binary-merkle-tree) of message share roots. Each message share root is [a subtree root in a row NMT](#arranging-available-data-into-shares). For rationale, see [rationale doc](../rationale/message_block_layout.md). +##### SignedTransactionDataPayForPadding + +| name | type | description | +| -------------------- | ------------------------------ | ------------------------------------------------------------ | +| `type` | `TransactionType` | Must be `TransactionType.PayForPadding`. | +| `messageNamespaceID` | [`NamespaceID`](#type-aliases) | Namespace ID of padding this transaction pays a fee for. | +| `messageSize` | `uint64` | Size of padding this transaction pays a fee for, in `byte`s. | + +Pays for the inclusion of a padding shares in the same block. Padding shares are used between real messages that are not tightly packed. For rationale, see [rationale doc](../rationale/message_block_layout.md). + ##### SignedTransactionDataCreateValidator | name | type | description | From c5cf93c85fd6907374ee5cda071cd0f623fa78fe Mon Sep 17 00:00:00 2001 From: John Adler Date: Tue, 24 Nov 2020 19:36:20 -0500 Subject: [PATCH 48/51] Add pay for padding tx state transition rule. --- specs/consensus.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/specs/consensus.md b/specs/consensus.md index b569f1c..71056a8 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -20,6 +20,7 @@ Consensus Rules - [`block.availableData.transactionData`](#blockavailabledatatransactiondata) - [SignedTransactionDataTransfer](#signedtransactiondatatransfer) - [SignedTransactionDataPayForMessage](#signedtransactiondatapayformessage) + - [SignedTransactionDataPayForPadding](#signedtransactiondatapayforpadding) - [SignedTransactionDataCreateValidator](#signedtransactiondatacreatevalidator) - [SignedTransactionDataBeginUnbondingValidator](#signedtransactiondatabeginunbondingvalidator) - [SignedTransactionDataUnbondValidator](#signedtransactiondataunbondvalidator) @@ -200,7 +201,6 @@ The following checks must be `true`: 1. `tx.type` == [`TransactionType.PayForMessage`](./data_structures.md#signedtransactiondata). 1. `tx.nonce` == `state.accounts[sender].nonce + 1`. -1. `uint(tx.messageNamespaceID) % 2` == `0`. In other words, the namespace ID interpreted as an unsigned integer is even. 1. The `ceil(tx.messageSize / SHARE_SIZE)` shares starting at index `wrappedTransactions.messageStartIndex` must: 1. Have namespace ID `tx.messageNamespaceID`. 1. `tx.messageShareCommitment` == computed as described [here](./data_structures.md#signedtransactiondatapayformessage). @@ -211,6 +211,17 @@ Apply the following to the state: state.accounts[sender].nonce += 1 ``` +#### SignedTransactionDataPayForPadding + +1. `tx.type` == [`TransactionType.PayForPadding`](./data_structures.md#signedtransactiondata). +1. The `ceil(tx.messageSize / SHARE_SIZE)` shares starting at index `wrappedTransactions.messageStartIndex` must: + 1. Have namespace ID `tx.messageNamespaceID`. + +Apply the following to the state: + +``` +``` + #### SignedTransactionDataCreateValidator The following checks must be `true`: From 152223916ce7409a55f332b1609f07fbb63367a8 Mon Sep 17 00:00:00 2001 From: John Adler Date: Thu, 26 Nov 2020 10:02:06 -0500 Subject: [PATCH 49/51] Update specs/consensus.md GRAFFITI_BYTES -> MAX_GRAFFITI_BYTES Co-authored-by: Ismail Khoffi --- specs/consensus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/consensus.md b/specs/consensus.md index 71056a8..8d52fd4 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -48,7 +48,7 @@ Consensus Rules | `AVAILABLE_DATA_ORIGINAL_SQUARE_MAX` | `uint64` | | `share` | Maximum number of rows/columns of the original data [shares](data_structures.md#share) in [square layout](data_structures.md#arranging-available-data-into-shares). | | `CHAIN_ID` | `uint64` | `1` | | Chain ID. Each chain assigns itself a (unique) ID. | | `GENESIS_COIN_COUNT` | `uint64` | `10**8` | `4u` | `(= 100000000)` Number of coins at genesis. | -| `GRAFFITI_BYTES` | `uint64` | `32` | `byte` | Maximum size of transaction graffiti, in bytes. | +| `MAX_GRAFFITI_BYTES` | `uint64` | `32` | `byte` | Maximum size of transaction graffiti, in bytes. | | `MAX_VALIDATORS` | `uint16` | `64` | | Maximum number of active validators. | | `NAMESPACE_ID_BYTES` | `uint64` | `8` | `byte` | Size of namespace ID, in bytes. | | `NAMESPACE_ID_MAX_RESERVED` | `uint64` | `254` | | Value of maximum reserved namespace ID (inclusive). 1 byte worth of IDs. | From 9a31d32acc2aab146f12668791cc6188245ace2d Mon Sep 17 00:00:00 2001 From: John Adler Date: Thu, 26 Nov 2020 11:21:09 -0500 Subject: [PATCH 50/51] Fix max namespace id reserved. --- specs/consensus.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/consensus.md b/specs/consensus.md index 8d52fd4..4b755db 100644 --- a/specs/consensus.md +++ b/specs/consensus.md @@ -48,10 +48,10 @@ Consensus Rules | `AVAILABLE_DATA_ORIGINAL_SQUARE_MAX` | `uint64` | | `share` | Maximum number of rows/columns of the original data [shares](data_structures.md#share) in [square layout](data_structures.md#arranging-available-data-into-shares). | | `CHAIN_ID` | `uint64` | `1` | | Chain ID. Each chain assigns itself a (unique) ID. | | `GENESIS_COIN_COUNT` | `uint64` | `10**8` | `4u` | `(= 100000000)` Number of coins at genesis. | -| `MAX_GRAFFITI_BYTES` | `uint64` | `32` | `byte` | Maximum size of transaction graffiti, in bytes. | +| `MAX_GRAFFITI_BYTES` | `uint64` | `32` | `byte` | Maximum size of transaction graffiti, in bytes. | | `MAX_VALIDATORS` | `uint16` | `64` | | Maximum number of active validators. | | `NAMESPACE_ID_BYTES` | `uint64` | `8` | `byte` | Size of namespace ID, in bytes. | -| `NAMESPACE_ID_MAX_RESERVED` | `uint64` | `254` | | Value of maximum reserved namespace ID (inclusive). 1 byte worth of IDs. | +| `NAMESPACE_ID_MAX_RESERVED` | `uint64` | `255` | | Value of maximum reserved namespace ID (inclusive). 1 byte worth of IDs. | | `SHARE_RESERVED_BYTES` | `uint64` | `1` | `byte` | Bytes reserved at the beginning of each [share](data_structures.md#share). Must be sufficient to represent `SHARE_SIZE`. | | `SHARE_SIZE` | `uint64` | `256` | `byte` | Size of transaction and message [shares](data_structures.md#share), in bytes. | | `STATE_SUBTREE_RESERVED_BYTES` | `uint64` | `1` | `byte` | Number of bytes reserved to identify state subtrees. | From 074cd26a7f69dc9051bfa7a615c1c1dcd860a7c5 Mon Sep 17 00:00:00 2001 From: John Adler Date: Fri, 27 Nov 2020 09:13:56 -0500 Subject: [PATCH 51/51] Fix missed graffiti bytes constant renaming. --- specs/data_structures.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/data_structures.md b/specs/data_structures.md index ee758ad..d26da38 100644 --- a/specs/data_structures.md +++ b/specs/data_structures.md @@ -78,7 +78,7 @@ Data Structures | `Amount` | `uint64` | | [`BlockID`](#blockid) | [HashDigest](#hashdigest) | | `FeeRate` | `uint64` | -| `Graffiti` | `byte[GRAFFITI_BYTES]` | +| `Graffiti` | `byte[MAX_GRAFFITI_BYTES]` | | [`HashDigest`](#hashdigest) | `byte[32]` | | `Height` | `uint64` | | `NamespaceID` | `byte[NAMESPACE_ID_BYTES]` |