docs: update shares specs (celestiaorg#1880)

* docs: extract shares page * ignore generated .html files * rename figures * update diagrams * update reserved bytes * fix: broken links * link to reserved namespace and x/blob * extract common padding specs * update parity section * overview -> abstract * add link to NMT spec * link to block * add Glossary * clarify Celestia state * move blob data to glossary * nit: use may * remove data withholding attack * draft: overview * nit: may -> MAY * move terms up and fill out overview * adopt more of template * Improve share sequence term * fill out share splitting * fix links * add note to clarify blob * add references section * Update specs/src/specs/shares.md Co-authored-by: Sanaz Taheri <[email protected]> * Update specs/src/specs/shares.md Co-authored-by: Sanaz Taheri <[email protected]> * add figure numbers * clarify share sequence length * blob -> transaction * denote field in diagram * remove duplicate portion of transaction shares * clarify share size * fix: lint b/c non-interactive => blob share commitment --------- Co-authored-by: Sanaz Taheri <[email protected]>
astriaorg · Jun 16, 2023 · 647ba7f · 647ba7f
1 parent 7989d88
commit 647ba7f
Show file tree

Hide file tree

Showing 16 changed files with 165 additions and 206 deletions.
diff --git a/.gitignore b/.gitignore
@@ -13,3 +13,4 @@ tmp/
 run.sh
 testing/e2e/networks/*/
 square/testdata
+**/*.html
diff --git a/pkg/shares/README.md b/pkg/shares/README.md
@@ -1,82 +1,3 @@
 # Shares
 
-Package shares provides primitives for splitting block data into shares and
-parsing shares back into block data.
-
-## Compact vs. Sparse
-
-There are two types of shares:
-
-1. Compact
-2. Sparse
-
-Compact shares can contain data from one or more unit (transactions or
-intermediate state roots). Sparse shares can contain data from zero or one blob.
-Compact shares and sparse shares are encoded differently. The motivation behind
-the distinction is that transactions and intermediate state roots are expected
-to have small lengths so they are encoded in compact shares to minimize the
-number of shares needed to store them. On the other hand, blobs are expected to
-be larger and have the desideratum that clients should be able to create proofs
-of blob inclusion. This desiradum is infeasible if client A's blob is encoded
-into a share with another client B's blob that is unknown to A. It follows that
-client A's blob is encoded into a share such that the contents can be determined
-by client A without any additional information. See [non-interactive default
-rules](https://celestiaorg.github.io/celestia-app/rationale/data_square_layout.html#non-interactive-default-rules)
-or [ADR 006](../../docs/architecture/adr-006-non-interactive-defaults.md) for
-more details.
-
-## Universal Prefix
-
-Both types of shares have a universal prefix. The first 1 byte of a share
-contains the namespace version. The next 28 bytes contain the namespace ID. The
-next one byte is an [InfoByte] that contains the share version and a sequence
-start indicator. If the sequence start indicator is `1` (i.e. this is the first
-share of a sequence) then the next 4 bytes contain a big endian uint32 of the
-sequence length. If the sequence start indicator is `0` (i.e. this is the
-continuation share of a sequence) then the remaing bytes of the share contain
-the share data.
-
-For the first share of a sequence:
-
-```text
-| namespace_version | namespace_id | info_byte | sequence_length | sequence_data            |
-| 1 byte            | 28 bytes     | 1 byte    | 4 bytes         | remaining bytes of share |
-```
-
-For continuation share of a sequence:
-
-```text
-| namespace_version | namespace_id | info_byte | sequence_data            |
-| 1 byte            | 28 bytes     | 1 byte    | remaining bytes of share |
-```
-
-The remaining bytes depend on the share type.
-
-## Compact Share Schema
-
-The four bytes after the universal prefix are reserved for
-the location in the share of the first unit of data that starts in this
-share.
-
-For the first compact share:
-
-```text
-| namespace_version | namespace_id | info_byte | sequence_length | location_of_first_unit | transactions or intermediate state roots |
-| 1 byte            | 28 bytes     | 1 byte    | 4 bytes         | 4 bytes                | remaining bytes of share                 |
-```
-
-For continuation compact share:
-
-```text
-| namespace_version | namespace_id | info_byte | location_of_first_unit | transactions or intermediate state roots |
-| 1 byte            | 28 bytes     | 1 byte    | 4 bytes                | remaining bytes of share                 |
-```
-
-Notes
-
-- All shares in a reserved namespace belong to one sequence.
-- Each unit (transaction or intermediate state root) in data is prefixed with a varint of the length of the unit.
-
-## Sparse Share Schema
-
-The remaining bytes contain blob data.
+See the specs for [shares](../../specs/src/specs/shares.md).
diff --git a/specs/src/README.md b/specs/src/README.md
@@ -2,6 +2,7 @@
 
 - [Specification](./specs/index.md)
   - [Data Structures](./specs/data_structures.md)
+  - [Shares](./specs/shares.md)
   - [Consensus](./specs/consensus.md)
   - [Block Proposer](./specs/block_proposer.md)
   - [Block Validity Rules](./specs/block_validity_rules.md)

diff --git a/specs/src/SUMMARY.md b/specs/src/SUMMARY.md
@@ -4,6 +4,7 @@
 
 - [Specification](./specs/index.md)
   - [Data Structures](./specs/data_structures.md)
+  - [Shares](./specs/shares.md)
   - [Consensus](./specs/consensus.md)
   - [Block Proposer](./specs/block_proposer.md)
   - [Block Validity Rules](./specs/block_validity_rules.md)

diff --git a/specs/src/specs/consensus.md b/specs/src/specs/consensus.md
@@ -30,7 +30,7 @@
 | `NAMESPACE_ID_MAX_RESERVED`             | `uint64` | `255`        |         | Value of maximum reserved namespace (inclusive). 1 byte worth of IDs.                                                                                               |
 | `SEQUENCE_BYTES`                        | `uint64` | `4`          | `byte`  | The number of bytes used to store the sequence length in the first share of a sequence                                                                              |
 | `SHARE_INFO_BYTES`                      | `uint64` | `1`          | `byte`  | The number of bytes used for [share](data_structures.md#share) information                                                                                          |
-| `SHARE_RESERVED_BYTES`                  | `uint64` | `4`          | `byte`  | The number of bytes used to store the location of the first unit in a compact share. Must be able to represent any integer up to and including `SHARE_SIZE - 1`.    |
+| `SHARE_RESERVED_BYTES`                  | `uint64` | `4`          | `byte`  | The number of bytes used to store the index of the first transaction in a transaction share. Must be able to represent any integer up to and including `SHARE_SIZE - 1`.    |
 | `SHARE_SIZE`                            | `uint64` | `512`        | `byte`  | Size of transaction and blob [shares](data_structures.md#share), in bytes.                                                                                          |
 | `STATE_SUBTREE_RESERVED_BYTES`          | `uint64` | `1`          | `byte`  | Number of bytes reserved to identify state subtrees.                                                                                                                |
 | `UNBONDING_DURATION`                    | `uint32` |              | `block` | Duration, in blocks, for unbonding a validator or delegation.                                                                                                       |
@@ -43,6 +43,7 @@
 |-------------------------------------|-------------|----------------------------------------------------------------|------------------------------------------------------------------------------------------------------|
 | `TRANSACTION_NAMESPACE`             | `Namespace` | `0x0000000000000000000000000000000000000000000000000000000001` | Transactions: requests that modify the state.                                                        |
 | `INTERMEDIATE_STATE_ROOT_NAMESPACE` | `Namespace` | `0x0000000000000000000000000000000000000000000000000000000002` | Intermediate state roots, committed after every transaction.                                         |
+| `PAY_FOR_BLOB_NAMESPACE`            | `Namespace` | `0x0000000000000000000000000000000000000000000000000000000004` | Namespace reserved for transactions that contain a PayForBlob.                                       |
 | `RESERVED_PADDING_NAMESPACE`        | `Namespace` | `0x00000000000000000000000000000000000000000000000000000000FF` | Padding after all reserved namespaces but before blobs.                                              |
 | `MAX_RESERVED_NAMESPACE`            | `Namespace` | `0x00000000000000000000000000000000000000000000000000000000FF` | Max reserved namespace is lexicographically the largest namespace that is reserved for protocol use. |
 | `TAIL_PADDING_NAMESPACE`            | `Namespace` | `0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFE` | Tail padding for blobs: padding after all blobs to fill up the original data square.                 |