Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: adds documentation to proofs package #3245

Merged
merged 44 commits into from
Apr 8, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
85995ce
docs: adds documentation to proofs package
rach-id Apr 2, 2024
70773d3
Merge branch 'main' into proofs-docs
rach-id Apr 2, 2024
c440d0a
docs: tidy details
rach-id Apr 2, 2024
620862f
Merge remote-tracking branch 'origin/proofs-docs' into proofs-docs
rach-id Apr 2, 2024
7c724dd
Update pkg/proof/README.md
rach-id Apr 2, 2024
ac9d26d
docs: fix range
rach-id Apr 2, 2024
89503a4
Update pkg/proof/README.md
rach-id Apr 2, 2024
5f229cb
Merge remote-tracking branch 'origin/proofs-docs' into proofs-docs
rach-id Apr 2, 2024
1ed2b10
docs: better important message
rach-id Apr 2, 2024
a794155
docs: rabbit review
rach-id Apr 2, 2024
c15ab0c
docs: rabbit review
rach-id Apr 2, 2024
20ee12a
docs: better images
rach-id Apr 3, 2024
78f48e7
docs: better wording
rach-id Apr 3, 2024
841580d
docs: better wording and better images
rach-id Apr 3, 2024
0658aad
docs: better wording
rach-id Apr 3, 2024
da8bc6b
Update pkg/proof/README.md
rach-id Apr 3, 2024
1617d61
Update pkg/proof/README.md
rach-id Apr 3, 2024
8576bb2
Merge branch 'main' into proofs-docs
rach-id Apr 3, 2024
f454b83
docs: BEFP
rach-id Apr 3, 2024
90f5880
docs: better important message
rach-id Apr 3, 2024
3a22d37
docs: share commitment link
rach-id Apr 3, 2024
3bcece8
Update pkg/proof/README.md
rach-id Apr 3, 2024
3f0a2eb
Update pkg/proof/README.md
rach-id Apr 3, 2024
0ca3dd2
Update pkg/proof/README.md
rach-id Apr 3, 2024
2deadb7
Update pkg/proof/README.md
rach-id Apr 3, 2024
02e21e3
Update pkg/proof/README.md
rach-id Apr 3, 2024
ddad6a5
Update pkg/proof/README.md
rach-id Apr 3, 2024
433789e
Update pkg/proof/README.md
rach-id Apr 3, 2024
8496a34
Update pkg/proof/README.md
rach-id Apr 3, 2024
c71ee77
Update pkg/proof/README.md
rach-id Apr 3, 2024
183f04e
Update pkg/proof/README.md
rach-id Apr 3, 2024
69863f9
Update pkg/proof/README.md
rach-id Apr 3, 2024
10a87cc
Update pkg/proof/README.md
rach-id Apr 3, 2024
c07f5fa
docs: SR3 to SR2
rach-id Apr 3, 2024
67f97ce
Merge remote-tracking branch 'origin/proofs-docs' into proofs-docs
rach-id Apr 3, 2024
4642e7e
Update pkg/proof/README.md
rach-id Apr 3, 2024
75c9054
docs: review feedback from @evan-forbes
rach-id Apr 5, 2024
f084fe4
Merge remote-tracking branch 'origin/proofs-docs' into proofs-docs
rach-id Apr 5, 2024
b8a5c0e
Merge branch 'main' into proofs-docs
rach-id Apr 5, 2024
b76698c
Update pkg/proof/README.md
rach-id Apr 8, 2024
74c1cc3
chore: remove new lines
rach-id Apr 8, 2024
8891754
chore: remove new lines
rach-id Apr 8, 2024
a38a4ed
chore: remove new lines
rach-id Apr 8, 2024
9b58805
Merge branch 'main' into proofs-docs
rach-id Apr 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 179 additions & 0 deletions pkg/proof/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
# Proofs
rach-id marked this conversation as resolved.
Show resolved Hide resolved
rach-id marked this conversation as resolved.
Show resolved Hide resolved

This package contains the proof queries for Celestia transactions/blobs.
rach-id marked this conversation as resolved.
Show resolved Hide resolved

# Details
rach-id marked this conversation as resolved.
Show resolved Hide resolved

This section will go over how the transactions/blobs inclusion proofs work.
rach-id marked this conversation as resolved.
Show resolved Hide resolved

## Requirements
rach-id marked this conversation as resolved.
Show resolved Hide resolved

To understand the proof mechanisms,
a good understanding of [binary merkle proofs](https://celestiaorg.github.io/celestia-app/specs/data_structures.html#merkle-trees) and [namespace merkle proofs](https://celestiaorg.github.io/celestia-app/specs/data_structures.html#namespace-merkle-tree) is required.
rach-id marked this conversation as resolved.
Show resolved Hide resolved
rach-id marked this conversation as resolved.
Show resolved Hide resolved

## PFB transaction

When creating a [PayForBlob](https://github.com/celestiaorg/celestia-app/blob/v1.0.0-rc2/proto/celestia/blob/v1/tx.proto#L16-L31) transaction,
rach-id marked this conversation as resolved.
Show resolved Hide resolved
the blob data is separated into a set of [shares](https://celestiaorg.github.io/celestia-app/specs/shares.html).
This set of shares is used to generate a *share commitment* which commits to that data.

Generating a *share commitment* starts with laying the shares into a [merkle mountain range](https://docs.grin.mw/wiki/chain-state/merkle-mountain-range/) structure.
rach-id marked this conversation as resolved.
Show resolved Hide resolved
For example, if the blob contains six shares, the following structure will be generated:

<img src="img/subtree_roots.png" alt="subtree roots generation from shares" width="500"/>

The blob shares are colored in green,
rach-id marked this conversation as resolved.
Show resolved Hide resolved
and are used
to generate a set of roots following the [merkle mountain range](https://docs.grin.mw/wiki/chain-state/merkle-mountain-range/) structure.
These roots are called *subtree roots*.
In the above case, we end up with two *subtree roots*: `SR1` and `SR2`.

Then, these *subtree roots* are used to generate the *share commitment*:

<img src="img/share_commitment.png" alt="share commitment generation from subtree roots" width="300"/>

The *share commitment* is the binary merkle root over the set of *subtree roots* generated from the shares.

## Share to share commitment inclusion

Using the above merkle trees,
share proofs can be generated to prove inclusion of a set of shares to the *share commitment*.

For example, if we want to prove the first two shares, we will need two merkle proofs:

<img src="img/share_to_share_commitment_inclusion.png" alt="shares to share commitment inclusion proof" width="500"/>

- The first merkle proof is a namespace merkle inclusion proof from the first two shares to the *subtree root*.
- The second merkle tree is a binary merkle inclusion proof from `SR1`,
rach-id marked this conversation as resolved.
Show resolved Hide resolved
which is the subtree root committing to the shares, to the *share commitment*.

Note: the nodes colored in brown are the inner nodes used to construct the inclusion proof.
rach-id marked this conversation as resolved.
Show resolved Hide resolved

## Square layout

Now that the *share commitment* is generated.
rach-id marked this conversation as resolved.
Show resolved Hide resolved
The transaction gets broadcasted to the Celestia network to be picked up by validators.
rach-id marked this conversation as resolved.
Show resolved Hide resolved
Once it's included in a block, the transaction, without the blob,
is placed in the [transaction namespace](https://celestiaorg.github.io/celestia-app/specs/namespace.html#reserved-namespaces)
and the blob is placed in the namespace specified in the PFB transaction.

If we use the same transaction from above,
where the blob consists of six shares, an example Celestia square containing the blob can look like this:

<img src="img/extended_square.png" alt="celestia extended square containing the blob shares" width="500"/>

The share range `[5, 11)`, colored in green, is the blob data.
rach-id marked this conversation as resolved.
Show resolved Hide resolved

### Row roots

To compute the Celestia block [data root](https://celestiaorg.github.io/celestia-app/specs/data_structures.html),
we first start by computing the row roots and column roots.
evan-forbes marked this conversation as resolved.
Show resolved Hide resolved

For example, if we want to compute the first row root:

<img src="img/row_root_1.png" alt="row root one generation" width="500"/>

We use the first row of shares to generate a namespace merkle root.
The first four are part of the square data, and the last four are parity shares.

Now if we take the second row, which contains four shares of the above blob:

<img src="img/row_root_2.png" alt="row root two generation" width="500"/>

The inner node in blue is the same as the *subtree root* `SR1`
rach-id marked this conversation as resolved.
Show resolved Hide resolved
generated in the [PFB transaction](#pfb-transaction) section.

Similarly, if we generate the third row, which contains the remaining two shares of the blob, and some other data:

<img src="img/row_root_3.png" alt="row root three generation" width="500"/>

We will see that the *subtree root* `SR2` is an inner node of the tree used to compute the third row root.

This property holds for all the subtree roots
computed for the blob data thanks to the [non-interactive defaults](https://celestiaorg.github.io/celestia-app/specs/data_square_layout.html) rules
when constructing the square.
This means
that it is possible to prove the inclusion of a blob to a set of row roots using the generated *share commitment*.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[question] why are there random newlines in the text? Can you please remove the newlines so it's easier to read

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's applying structural wrapping. I've seen the convention in the team to use structural wrapping in the docs and everywhere, and goland provides automatic structural wrapping. I am a fan of long lines btw :D

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is structural wrapping?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a feature that automatically formats texts based on it structure, for example it adds a new line after , or .

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we disable that feature, it makes this content difficult to read. Also why did it add a newline after "means" here:

This means
that it is possible to prove the inclusion of a blob to a set of row roots using the generated *share commitment*.

We will discuss this proof below.

### Column roots

Similar to row roots, column roots are the namespace merkle root of the set of shares contained in the column:

<img src="img/col_root_1.png" alt="column root one generation" width="427"/>

### Data root

The Celestia block [data root](https://celestiaorg.github.io/celestia-app/specs/data_structures.html) is computed
by generating a binary merkle root over the set of rows roots and column roots of the square:
rach-id marked this conversation as resolved.
Show resolved Hide resolved

<img src="img/data_root.png" alt="data root generation" width="443"/>

This allows inclusion proofs of the shares to the data root which will be discussed below.
rach-id marked this conversation as resolved.
Show resolved Hide resolved

## Inclusion proof

### Prove share inclusion to data root

To prove that a share was included in a Celestia block,
we can create an inclusion proof from the share to the data root.
This proof consists of the following:

#### Share to row root namespace merkle inclusion proof:

First, we prove inclusion of the share in question to the row root it belongs to.
If we take, for example, share three, its inclusion proof can be constructed using the inner nodes in brown:

<img src="img/share_to_row_root_proof.png" alt="share to row root proof" width="443"/>

Now, if we have this inclusion proof, and we verify it against the `RowRoot1`, the proof would be valid.

#### Row root to data root inclusion proof

Now that we proved the share to the row root,
we will need to complete the proof with a row root to the data root inclusion proof.
This means that we will be proving that the row root commits to the share, and the data root commits to the row root.
Since share three is committed to by `RowRoot1`, we will be proving that row root to the data root.

<img src="img/row_root_to_data_root_proof.png" alt="row root to data root proof" width="443"/>

The inner nodes in brown are the nodes needed to construct the merkle inclusion proof.
rach-id marked this conversation as resolved.
Show resolved Hide resolved

So, if we have access to these two proofs,
the [share to row root inclusion proof](#share-to-row-root-namespace-merkle-inclusion-proof) and [row root to data root inclusion proof](#row-root-to-data-root-inclusion-proof),
we can successfully prove inclusion of share three to the Celestia data root.

### Prove share commitment inclusion to data root

To prove that a blob,
referenced by its *share commitment*, was included in a Celestia block, we need the following proofs:

#### Subtree roots inclusion proof to the data root

As we've seen in the [row roots generation](#row-roots) section,
the subtree roots are inner nodes of the namespace merkle tree used to generate the row root.
This means that it's possible to create an inclusion proof of a subtree root to the row root:

<img src="img/subtree_root_to_row_root.png" alt="subtree root to row root proof" width="443"/>

The inner nodes in brown would be used to construct the `SR2` node inclusion proof to the `RowRoot3`.

After having this proof, we will need to prove the `RowRoot3` inclusion to the data root.
This is described in the [row root to data root inclusion proof](#row-root-to-data-root-inclusion-proof) section.

With this, we would be able to prove that `SR3` was committed to by a Celestia data root,
i.e., was included in a Celestia block.

#### Subtree roots inclusion proof to the *share commitment*

Now that we can prove `SR2` inclusion to `RowRoot3`,
what's left is proving that `SR2` was committed to by the *share commitment*.

Proving inclusion of `SR1` or `SR2` to the *share commitment* goes back
to creating a binary merkle inclusion proof to the *share commitment*:

<img src="img/share_commitment.png" alt="share commitment generation from subtree roots" width="300"/>

Now that we managed to prove that `SR1` and `SR2` were both committed to by the Celestia data root, and that the *share commitment* was generated using `SR1` and `SR2`, then, we would have proven that the *share commitment* was committed to by the Celestia data root, which means that **the blob data that generated the *share commitment* was included in a Celestia block**.

Note: Currently, the generation/verification of these proofs is still not supported.
rach-id marked this conversation as resolved.
Show resolved Hide resolved
Binary file added pkg/proof/img/col_root_1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pkg/proof/img/data_root.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pkg/proof/img/extended_square.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pkg/proof/img/row_root_1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pkg/proof/img/row_root_2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pkg/proof/img/row_root_3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pkg/proof/img/row_root_to_data_root_proof.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pkg/proof/img/share_commitment.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pkg/proof/img/share_to_row_root_proof.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pkg/proof/img/subtree_root_to_row_root.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pkg/proof/img/subtree_roots.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading