Add state representation

rationale/distributing_rewards.md

@@ -0,0 +1,54 @@
+Rationale: Distributing Rewards and Penalties
+===
+
+- [Rationale: Distributing Rewards and Penalties](#rationale-distributing-rewards-and-penalties)
+- [Preamble](#preamble)
+- [Distribution Scheme](#distribution-scheme)
+  - [State-Efficient Implementation](#state-efficient-implementation)
+
+# Preamble
+
+Due to the requirement that all incorrect state transitions on LazyLedger be provable with a [compact fraud proof](https://arxiv.org/abs/1809.09044) that is cheap enough to verify within a smart contract on a remote chain (e.g. Ethereum), computing how rewards and penalties are distributed must involve no iterations. To understand why, let us consider the following desiderata in a staking system:
+1. In-protocol stake delegation: this makes it easier for users to participate in the consensus process, and reduces reliance on custodial staking services.
+1. In-protocol enforcement of proper distribution of rewards and penalities to delegators: rewards and penalties collected by validators should be distributed to delegators trustlessly.
+
+Naively, rewards and penalties (henceforth referred to collectively as "rewards", since penalties are simply negative rewards) can be distributed immediately. For example, when a validator produces a new block and is entitled to collecting transaction fees, these fees can be distributed to every single account delegating stake to this validator. This requires iterating over potentially a huge number of state elements for a single state transition (i.e. transaction), which is computationally expensive. The specific problem is that it would be infeasible to prove that such a state transition was _incorrect_ (i.e. with a fraud proof) within the execution system of a remote blockchain (i.e. with a smart contract).
+
+This forms the primary motivation of the scheme discussed here: a mechanism for distributing rewards that is state-efficient and requires no iteration over state elements for any state transition.
+
+# Distribution Scheme
+
+The scheme presented here is an incarnation of Cosmos' [F1 fee distribution scheme](https://github.com/cosmos/cosmos-sdk/blob/master/docs/spec/_proposals/f1-fee-distribution/f1_fee_distr.pdf). F1 has the nice property of being approximation-free and, with proper implementation details, can be highly efficient with state usage and completely iteration-free in all cases.
+
+Naively, when considering a single block, the reward that should be given to a delegator with stake $x$, who's delegating to a validator with total voting power $n$, whose reward in that block is $T$, is:
+
+$$
+\text{naive reward} = x \frac{T}{n}
+$$
+
+In other words, the voting power contributed by the delegator multiplied by the _reward rate_, i.e. the rewards per unit of voting power. We note that if the total voting power of a validator remains constant forever, then the above equations hold and is approximation-free. However, changes to the total voting power need to be accounted for.
+
+Blocks between two changes to a validator's voting power (i.e. whenever a user delegates or undelegates stake) are a _period_. Every time a validator's voting power changes (i.e. a new period $f$ begins), an entry $Entry_f$ for this period is saved in state, which records _the reward rate up to the beginning of_ $f$. This is simply the sum of the reward rate up to the beginning of previous period $f-1$ and the reward rate of the period $f$ itself:
+
+$$
+Entry_f = \begin{cases}
+    0 & f = 0 \\
+    Entry_{f-1} + \frac{T_f}{n_f} & f > 0 \\
+\end{cases}
+$$
+
+Finally, the raw reward for a delegation is simply proportional to the difference in entries between the period where undelegation ended ($f$) and where it began ($k$).
+
+$$
+\text{reward} = x (Entry_f - Entry_k)
+$$
+
+This raw reward can be scaled by additional factors, such as commissions or slashing penalties.
+
+## State-Efficient Implementation
+
+The F1 paper doesn't specify where entries are stored in state, but the understanding is that they are placed in independent state elements. This has the downside of requiring multiple Merkle branches to prove the inclusion of entries for e.g. fraud proofs. We can improve on this by leveraging a specific property of entries, namely that each entry is only used in exactly two cases:
+1. To compute the next entry.
+1. To compute the reward of a delegator.
+
+Intuitively, after having being used twice, an entry can be pruned from the state. We can make use of this by storing only the latest entry with its respective validator object, and a copy of the two entries each delegation needs with the delegation object. By storing entries directly with the objects that require them, state transitions can be statelessly validated without extra inclusion proofs.

specs/consensus.md

@@ -11,6 +11,8 @@ Consensus Rules
 - [Leader Selection](#leader-selection)
 - [Fork Choice](#fork-choice)
 - [Block Validity](#block-validity)
+  - [State Transitions](#state-transitions)
+    - [Validators and Delegations](#validators-and-delegations)
   - [Formatting](#formatting)
   - [Availability](#availability)
 
@@ -37,6 +39,8 @@ Consensus Rules
 | `SHARE_RESERVED_BYTES`                | `uint64` | `1`     | `byte`  | Bytes reserved at the beginning of each [share](data_structures.md#share). Must be sufficient to represent `SHARE_SIZE`.                                    |
 | `AVAILABLE_DATA_ORIGINAL_SQUARE_SIZE` | `uint64` |         | `share` | Number of rows/columns of the original data [shares](data_structures.md#share) in [square layout](data_structures.md#arranging-available-data-into-shares). |
 | `GENESIS_COIN_COUNT`                  | `uint64` | `10**8` | `4u`    | `(= 100000000)` Number of coins at genesis.                                                                                                                 |
+| `UNBONDING_DURATION`                  | `uint32` |         | `block` | Duration, in blocks, for unbonding a validator or delegation.                                                                                               |
+| `MAX_VALIDATORS`                      | `uint16` | `64`    |         | Maximum number of active validators.                                                                                                                        |
 
 ## Types
 
@@ -66,6 +70,76 @@ Consensus Rules
 
 # Block Validity
 
+## State Transitions
+
+### Validators and Delegations
+
+A transaction `tx` that requests a new validator initializes the [Validator](data_structures.md#validator) fields of that account as follows:
+| name              | value                    |
+| ----------------- | ------------------------ |
+| `status`          | `ValidatorStatus.Queued` |
+| `stakedBalance`   | `tx.amount`              |
+| `commissionRate`  | `tx.commissionRate`      |
+| `delegatedCount`  | `0`                      |
+| `votingPower`     | `tx.amount`              |
+| `pendingRewards`  | `0`                      |
+| `latestEntry`     | `PeriodEntry(0)`         |
+| `unbondingHeight` | `0`                      |
+| `isSlashed`       | `false`                  |
+
+At the end of a block at the end of an epoch, the top `MAX_VALIDATORS` validators by voting power are or become active. For newly-bonded validators, their status is changed to bonded and height values initialized.
+| name     | value                    |
+| -------- | ------------------------ |
+| `status` | `ValidatorStatus.Bonded` |
+
+For validators that were bonded but are no longer (either by being outside the top validators or through a transaction that requests unbonding), they begin unbonding.
+| name              | value                       |
+| ----------------- | --------------------------- |
+| `status`          | `ValidatorStatus.Unbonding` |
+| `unbondingHeight` | `block.height + 1`          |
+
+Once an unbonding validator has waited at least `UNBONDING_DURATION` blocks, they can be unbonded, collecting their reward:
+| name            | value                                 |
+| --------------- | ------------------------------------- |
+| `status`        | `ValidatorStatus.Unbonded`            |
+| `stakedBalance` | `0`                                   |
+| `votingPower`   | `old.votingPower - old.stakedBalance` |
+
+Every time a bonded validator's voting power changes (i.e. when a delegation is added or removed), or when a validator begins unbonding, the rewards per unit of voting power accumulated so far are calculated:
+| name             | value                                                    |
+| ---------------- | -------------------------------------------------------- |
+| `pendingRewards` | `0`                                                      |
+| `latestEntry`    | `old.latestEntry + old.pendingRewards / old.votingPower` |
+
+A transaction `tx` that requests a new delegation first updates the target validator's voting power:
+| name             | value                         |
+| ---------------- | ----------------------------- |
+| `delegatedCount` | `old.delegatedCount + 1`      |
+| `votingPower`    | `old.votingPower + tx.amount` |
+
+then initializes the [Delegation](data_structures.md#delegation) field of that account as follows:
+| name              | value                     |
+| ----------------- | ------------------------- |
+| `status`          | `DelegationStatus.Bonded` |
+| `validator`       | `tx.validator`            |
+| `stakedBalance`   | `tx.amount`               |
+| `beginEntry`      | `validator.latestEntry`   |
+| `endEntry`        | `PeriodEntry(0)`          |
+| `unbondingHeight` | `0`                       |
+
+A transaction `tx` that requests withdrawing a delegation first updates the delegation field:
+| name              | value                        |
+| ----------------- | ---------------------------- |
+| `status`          | `DelegationStatus.Unbonding` |
+| `endEntry`        | `validator.latestEntry`      |
+| `unbondingHeight` | `block.height + 1`           |
+
+then updates the target validator's voting power:
+| name             | value                                      |
+| ---------------- | ------------------------------------------ |
+| `delegatedCount` | `old.delegatedCount - 1`                   |
+| `votingPower`    | `old.votingPower - delegation.votingPower` |
+
 ## Formatting
 
 Leaves in the message [Namespace Merkle Tree](data_structures.md#namespace-merkle-tree) must be ordered lexicographically by namespace ID.

specs/data_structures.md

@@ -43,6 +43,11 @@ Data Structures
   - [MessageData](#messagedata)
     - [Message](#message)
 - [State](#state)
+  - [Account](#account)
+  - [PeriodEntry](#periodentry)
+  - [Validator](#validator)
+  - [Delegation](#delegation)
+  - [Decimal](#decimal)
 - [Consensus Parameters](#consensus-parameters)
 
 # Data Structures Overview
@@ -475,7 +480,91 @@ enum VoteType : uint8_t {
 
 # State
 
-TODO validator set repr
+| name            | type                      | description                  |
+| --------------- | ------------------------- | ---------------------------- |
+| `accountsRoot`  | [HashDigest](#hashdigest) | Merkle root of account tree. |
+| `numValidators` | `uint32`                  | Number of active validators. |
+
+The state of the LazyLedger chain contains only account balances and the validator set (which is extra metadata on top of the plain account balances).
+
+One unified [Sparse Merkle Trees](#sparse-merkle-tree) is maintained for both account account balances and validator metadata, the _accounts tree_. The final state root is computed as the [hash](#hashdigest) of the accounts tree root and number of active validators. The latter is necessary to ensure light nodes can determine the entire validator set from a single state root commitment.
+
+## Account
+
+| name             | type                      | description                                                                                 |
+| ---------------- | ------------------------- | ------------------------------------------------------------------------------------------- |
+| `balance`        | `uint64`                  | Coin balance.                                                                               |
+| `nonce`          | `uint64`                  | Account nonce. Every outgoing transaction from this account increments the nonce.           |
+| `isValidator`    | `bool`                    | Whether this account is a validator or not. Mutually exclusive with `isDelegating`.         |
+| `validatorInfo`  | [Validator](#validator)   | _Optional_, only if `isValidator` is set. Validator info.                                   |
+| `isDelegating`   | `bool`                    | Whether this account is delegating its stake or not. Mutually exclusive with `isValidator`. |
+| `delegationInfo` | [Delegation](#delegation) | _Optional_, only if `isDelegating` is set. Delegation info.                                 |
+
+In the accounts tree, accounts (i.e. leaves) are keyed by the [hash](#hashdigest) of their [address](#address).
+
+## PeriodEntry
+
+| name         | type     | description                                                   |
+| ------------ | -------- | ------------------------------------------------------------- |
+| `rewardRate` | `uint64` | Rewards per unit of voting power accumulated so far, in `1u`. |
+
+For explanation on entries, see the [reward distribution rationale document](../rationale/distributing_rewards.md).
+
+## Validator
+
+```C++
+enum ValidatorStatus : uint8_t {
+    Queued = 1,
+    Bonded = 2,
+    Unbonding = 3,
+    Unbonded = 4,
+};
+```
+
+| name              | type                        | description                                                                            |
+| ----------------- | --------------------------- | -------------------------------------------------------------------------------------- |
+| `status`          | `ValidatorStatus`           | Status of this validator.                                                              |
+| `stakedBalance`   | `uint64`                    | Validator's personal staked balance, in `4u`.                                          |
+| `commissionRate`  | [Decimal](#decimal)         | Commission rate.                                                                       |
+| `delegatedCount`  | `uint32`                    | Number of accounts delegating to this validator.                                       |
+| `votingPower`     | `uint64`                    | Total voting power as staked balance + delegated stake, in `4u`.                       |
+| `pendingRewards`  | `uint64`                    | Rewards collected so far this period, in `1u`.                                         |
+| `latestEntry`     | [PeriodEntry](#periodentry) | Latest entry, used for calculating reward distribution.                                |
+| `unbondingHeight` | `uint64`                    | Block height validator began unbonding.                                                |
+| `isSlashed`       | `bool`                      | If this validator has been slashed or not.                                             |
+| `slashRate`       | [Decimal](#decimal)         | _Optional_, only if `isSlashed` is set. Rate at which this validator has been slashed. |
+
+Validator objects represent all the information needed to be keep track of a validator. Validators have four statuses:
+1. `Queued`: This validator has entered the queue to become an active validator. Once the next validator set transition occurs, if this validator has sufficient voting power (including its own stake and stake delegated to it) to be in the top `MAX_VALIDATORS` validators by voting power, it will become an active, i.e. `Bonded` validator. Until bonded, this validator can immediately exit the queue.
+1. `Bonded`: This validator is active and bonded. It can propose new blocks and vote on proposed blocks. Once bonded, an active validator must go through an unbonding process until its stake can be freed.
+1. `Unbonding`: This validator is in the process of unbonding, which can be voluntary (the validator decided to stop being an active validator) or forced (the validator committed a slashable offence and was kicked from the active validator set). Validators will remain in this status for at least `UNBONDING_DURATION` blocks, and while unbonding may still be slashed.
+1. `Unbonded`: This validator has completed its unbonding and has withdrawn its stake. The validator object will remain in this status until `delegatedCount` reaches zero, at which point it is destroyed.
+
+## Delegation
+
+```C++
+enum DelegationStatus : uint8_t {
+    Bonded = 1,
+    Unbonding = 2,
+};
+```
+
+| name              | type                        | description                                         |
+| ----------------- | --------------------------- | --------------------------------------------------- |
+| `status`          | `DelegationStatus`          | Status of this delegation.                          |
+| `validator`       | [Address](#address)         | The validator being delegating to.                  |
+| `stakedBalance`   | `uint64`                    | Delegated stake, in `4u`.                           |
+| `beginEntry`      | [PeriodEntry](#periodentry) | Entry when delegation began.                        |
+| `endEntry`        | [PeriodEntry](#periodentry) | Entry when delegation ended (i.e. began unbonding). |
+| `unbondingHeight` | `uint64`                    | Block height delegation began unbonding.            |
+
+Delegation objects represent a delegation. They have two statuses:
+1. `Bonded`: This delegation is enabled for a `Queued` _or_ `Bonded` validator. Delegations to a `Queued` validator can be withdrawn immediately, while delegations for a `Bonded` validator must be unbonded first.
+1. `Unbonding`: This delegation is unbonding. It will remain in this status for at least `UNBONDING_DURATION` blocks, and while unbonding may still be slashed. Once the unbonding duration has expired, the delegation can be withdrawn.
+
+## Decimal
+
+TODO define a format for numbers in the range `[0,1]`
 
 # Consensus Parameters