docs: complete the Treestate RFC documentation

Author: VolodymyrBg

book/src/dev/rfcs/drafts/0005-treestate.md

@@ -27,7 +27,7 @@ finalized state (for Sprout).
 # Definitions
 [definitions]: #definitions
 
-TODO: split up these definitions into common, Sprout, Sapling, and possibly Orchard sections
+## Common Definitions
 
 Many terms used here are defined in the [Zcash Protocol Specification](https://zips.z.cash/protocol/protocol.pdf)
 
@@ -63,27 +63,31 @@ of the Merkle tree’s hash function.  Since the `Nullifier` set is always updat
 together with the `NoteCommitment` tree, this also identifies a particular state
 of the associated `Nullifier` set.
 
+## Sprout Definitions
+
+**joinsplit**: A shielded transfer that can spend Sprout `Note`s and transparent
+value, and create new Sprout `Note`s and transparent value, in one Groth16 proof
+statement.
+
+## Sapling Definitions
+
 **spend descriptions**: A shielded Sapling transfer that spends a `Note`. Includes
 an anchor of some previous `Block`'s `NoteCommitment` tree.
 
 **output descriptions**: A shielded Sapling transfer that creates a
 `Note`. Includes the u-coordinate of the `NoteCommitment` itself.
 
+## Orchard Definitions
+
 **action descriptions**: A shielded Orchard transfer that spends and/or creates a `Note`. 
 Does not include an anchor, because that is encoded once in the `anchorOrchard` 
 field of a V5 `Transaction`.
 
 
-
-**joinsplit**: A shielded transfer that can spend Sprout `Note`s and transparent
-value, and create new Sprout `Note`s and transparent value, in one Groth16 proof
-statement.
-
-
 # Guide-level explanation
 [guide-level-explanation]: #guide-level-explanation
 
-TODO: split into common, Sprout, Sapling, and probably Orchard sections
+## Common Processing for All Protocols
 
 As `Block`s are validated, the `NoteCommitment`s revealed by all the transactions
 within that block are used to construct `NoteCommitmentTree`s, with the
@@ -101,45 +105,33 @@ Sapling anchors). Sapling block validation includes comparing the specified
 FinalSaplingRoot in its block header to the root of the Sapling `NoteCommitment`
 tree that we have just computed to make sure they match.
 
+## Sprout Processing
+
+For Sprout, we must compute/update interstitial `NoteCommitmentTree`s between
+`JoinSplit`s that may reference an earlier one's root as its anchor. If we do
+this at the transaction layer, we can iterate through all the `JoinSplit`s and
+compute the Sprout `NoteCommitmentTree` and nullifier set similar to how we do
+the Sapling ones as described above, but at each state change (ie,
+per-`JoinSplit`) we note the root and cache it for lookup later. As the
+`JoinSplit`s are validated without context, we check for its specified anchor
+amongst the interstitial roots we've already calculated (according to the spec,
+these interstitial roots don't have to be finalized or the result of an
+independently validated `JoinSplit`, they just must refer to any prior `JoinSplit`
+root in the same transaction). So we only have to wait for our previous root to
+be computed via any of our candidates, which in the worst case is waiting for
+all of them to be computed for the last `JoinSplit`. If our `JoinSplit`s defined
+root pops out, that `JoinSplit` passes that check.
+
+## Sapling Processing
+
 As the transactions within a block are parsed, Sapling shielded transactions
 including `Spend` descriptions and `Output` descriptions describe the spending and
-creation of Zcash Sapling notes, and JoinSplit-on-Groth16 descriptions to
-transfer/spend/create Sprout notes and transparent value. `JoinSplit` and `Spend`
-descriptions specify an anchor, which references a previous `NoteCommitment` tree
-root: for `Spend`s, this is a previous block's anchor as defined in their block
-header, for `JoinSplit`s, it may be a previous block's anchor or the root
-produced by a strictly previous `JoinSplit` description in its transaction. For
-`Spend`s, this is convenient because we can query our state service for
+creation of Zcash Sapling notes. `Spend` descriptions specify an anchor, which references a previous 
+`NoteCommitment` tree root. This is a previous block's anchor as defined in their block header. 
+This is convenient because we can query our state service for
 previously finalized Sapling block anchors, and if they are found, then that
 [consensus check](https://zips.z.cash/protocol/canopy.pdf#spendsandoutputs) has
-been satisfied and the `Spend` description can be validated independently. For
-`JoinSplit`s, if it's not a previously finalized block anchor, it must be the
-treestate anchor of previous `JoinSplit` in this transaction, and we have to wait
-for that one to be parsed and its root computed to check that ours is
-valid. Luckily, it can only be a previous `JoinSplit` in this transaction, and is
-[usually the immediately previous one](zcashd), so the set of candidate anchors
-is smaller for earlier `JoinSplit`s in a transaction, but larger for the later
-ones. For these `JoinSplit`s, they can be validated independently of their
-anchor's finalization status as long as the final check of the anchor is done,
-when available, such as at the Transaction level after all the `JoinSplit`s have
-finished validating everything that can be validated without the context of
-their anchor's finalization state.
-
-So for each transaction, for both `Spend` descriptions and `JoinSplit`s, we can
-preemptively try to do our consensus check by looking up the anchors in our
-finalized set first. For `Spend`s, we then trigger the remaining validation and
-when that finishes we are full done with those. For `JoinSplit`s, the anchor
-state check may pass early if it's a previous block Sprout `NoteCommitment` tree
-root, but it may fail because it's an earlier `JoinSplit`s root instead, so once
-the `JoinSplit` validates independently of the anchor, we wait for all candidate
-previous `JoinSplit`s in that transaction finish validating before doing the
-anchor consensus check again, but against the output treestate roots of earlier
-`JoinSplit`s.
-
-Both Sprout and Sapling `NoteCommitment` trees must be computed for the whole
-block to validate. For Sprout, we need to compute interstitial treestates in
-between `JoinSplit`s in order to do the final consensus check for each/all
-`JoinSplit`s, not just for the whole block, as in Sapling.
+been satisfied and the `Spend` description can be validated independently.
 
 For Sapling, at the block layer, we can iterate over all the transactions in
 order and if they have `Spend`s and/or `Output`s, we update our Nullifer set for
@@ -149,31 +141,29 @@ them as leaves in positions according to their order as they appear transaction
 to transaction, output to output, in the block. This can be done independent of
 the transaction validations. When the Sapling transactions are all validated,
 the `NoteCommitmentTree` root should be computed: this is the anchor for this
-block. For Sapling and Blossom blocks, we need to check that this root matches
+block.
+
+## Anchor Validation Across Network Upgrades
+
+For Sapling and Blossom blocks, we need to check that this root matches
 the `RootHash` bytes in this block's header, as the `FinalSaplingRoot`. Once all
 other consensus and validation checks are done, this will be saved down to our
 finalized state to our `sapling_anchors` set, making it available for lookup by
 other Sapling descriptions in future transactions.
-TODO: explain Heartwood, Canopy, NU5 rule variants around anchors.
-For Sprout, we must compute/update interstitial `NoteCommitmentTree`s between
-`JoinSplit`s that may reference an earlier one's root as its anchor. If we do
-this at the transaction layer, we can iterate through all the `JoinSplit`s and
-compute the Sprout `NoteCommitmentTree` and nullifier set similar to how we do
-the Sapling ones as described above, but at each state change (ie,
-per-`JoinSplit`) we note the root and cache it for lookup later. As the
-`JoinSplit`s are validated without context, we check for its specified anchor
-amongst the interstitial roots we've already calculated (according to the spec,
-these interstitial roots don't have to be finalized or the result of an
-independently validated `JoinSplit`, they just must refer to any prior `JoinSplit`
-root in the same transaction). So we only have to wait for our previous root to
-be computed via any of our candidates, which in the worst case is waiting for
-all of them to be computed for the last `JoinSplit`. If our `JoinSplit`s defined
-root pops out, that `JoinSplit` passes that check.
 
-To finalize the block, the Sprout and Sapling treestates are the ones resulting
-from the last transaction in the block, and determines the Sprout and Sapling
-anchors that will be associated with this block as we commit it to our finalized
-state. The Sprout and Sapling nullifiers revealed in the block will be merged
+In Heartwood and Canopy, the rules for final Sapling roots are modified to support empty blocks by allowing an empty subtree hash instead of requiring the root to match the previous block's final Sapling root when there are no Sapling transactions.
+
+In NU5, the rules are further extended to include Orchard note commitment trees, with similar logic applied to the `anchorOrchard` field in V5 transactions.
+
+## Orchard Processing
+
+For Orchard, similar to Sapling, action descriptions can spend and create notes. The anchor is specified at the transaction level in the `anchorOrchard` field of a V5 transaction. The process follows similar steps to Sapling for validation and inclusion in blocks.
+
+## Block Finalization
+
+To finalize the block, the Sprout, Sapling, and Orchard treestates are the ones resulting
+from the last transaction in the block, and determines the respective anchors that will be associated with this block as we commit it to our finalized
+state. The nullifiers revealed in the block will be merged
 with the existing ones in our finalized state (ie, it should strictly grow over
 time).
 
@@ -184,14 +174,14 @@ time).
 - When finalizing a block, the finalized tip is updated with a serialization of the latest Orchard Note Commitment Tree. (The previous tree should be deleted as part of the same database transaction.)
 - Each non-finalized chain gets its own copy of the Orchard note commitment tree, cloned from the note commitment tree of the finalized tip or fork root.
 - When a block is added to a non-finalized chain tip, the Orchard note commitment tree is updated with the note commitments from that block.
-- When a block is rolled back from a non-finalized chain tip... (TODO)
+- When a block is rolled back from a non-finalized chain tip, the Orchard tree state is restored to its previous state before the block was added. This involves either keeping a reference to the previous state or recalculating from the fork point.
 
 ### Sapling
 - There is a single copy of the latest Sapling Note Commitment Tree for the finalized tip.
 - When finalizing a block, the finalized tip is updated with a serialization of the Sapling Note Commitment Tree. (The previous tree should be deleted as part of the same database transaction.)
 - Each non-finalized chain gets its own copy of the Sapling note commitment tree, cloned from the note commitment tree of the finalized tip or fork root.
 - When a block is added to a non-finalized chain tip, the Sapling note commitment tree is updated with the note commitments from that block.
-- When a block is rolled back from a non-finalized chain tip... (TODO)
+- When a block is rolled back from a non-finalized chain tip, the Sapling tree state is restored to its previous state, similar to the Orchard process. This involves either maintaining a history of tree states or recalculating from the fork point.
 
 ### Sprout
 - Every finalized block stores a separate copy of the Sprout note commitment tree (😿), as of that block.
@@ -205,23 +195,84 @@ We can't just compute a fresh tree with just the note commitments within a block
 # Reference-level explanation
 [reference-level-explanation]: #reference-level-explanation
 
+The implementation involves several key components:
+
+1. **Incremental Merkle Trees**: We use the `incrementalmerkletree` crate to implement the note commitment trees for each shielded pool.
+
+2. **Nullifier Storage**: We maintain nullifier sets in RocksDB to efficiently check for duplicates.
+
+3. **Tree State Management**: 
+   - For finalized blocks, we store the tree states in RocksDB.
+   - For non-finalized chains, we keep tree states in memory.
+
+4. **Anchor Verification**:
+   - For Sprout: we check anchors against our stored Sprout tree roots.
+   - For Sapling: we compare the computed root against the block header's `FinalSaplingRoot`.
+   - For Orchard: we validate the `anchorOrchard` field in V5 transactions.
+
+5. **Re-insertion Prevention**: Our implementation should prevent re-inserts of keys that have been deleted from the database, as this could lead to inconsistencies. The state service tracks deletion events and validates insertion operations accordingly.
 
 # Drawbacks
 [drawbacks]: #drawbacks
 
+1. **Storage Requirements**: Storing separate tree states (especially for Sprout) requires significant disk space.
+
+2. **Performance Impact**: Computing and verifying tree states can be computationally expensive, potentially affecting sync performance.
 
+3. **Implementation Complexity**: Managing multiple tree states across different protocols adds complexity to the codebase.
+
+4. **Fork Handling**: Maintaining correct tree states during chain reorganizations requires careful handling.
 
 # Rationale and alternatives
 [rationale-and-alternatives]: #rationale-and-alternatives
 
+We chose this approach because:
+
+1. **Protocol Compatibility**: Our implementation follows the Zcash protocol specification requirements for handling note commitment trees and anchors.
+
+2. **Performance Optimization**: By caching tree states, we avoid recomputing them for every validation operation.
+
+3. **Memory Efficiency**: For non-finalized chains, we only keep necessary tree states in memory.
+
+4. **Scalability**: The design scales with chain growth by efficiently managing storage requirements.
+
+Alternative approaches considered:
+
+1. **Recompute Trees On-Demand**: Instead of storing tree states, recompute them when needed. This would save storage but significantly impact performance.
+
+2. **Single Tree State**: Maintain only the latest tree state and recompute for historical blocks. This would simplify implementation but make historical validation harder.
+
+3. **Full History Storage**: Store complete tree states for all blocks. This would optimize validation speed but require excessive storage.
 
 # Prior art
 [prior-art]: #prior-art
 
+1. **Zcashd**: Uses similar concepts but with differences in implementation details, particularly around storage and concurrency.
+
+2. **Lightwalletd**: Provides a simplified approach to tree state management focused on scanning rather than full validation.
+
+3. **Incrementalmerkletree Crate**: Our implementation leverages this existing Rust crate for efficient tree management.
 
 # Unresolved questions
 [unresolved-questions]: #unresolved-questions
 
+1. **Optimization Opportunities**: Are there further optimizations we can make to reduce storage requirements while maintaining performance?
+
+2. **Root Storage**: Should we store the `Root` hash in `sprout_note_commitment_tree`, and use it to look up the complete tree state when needed?
+
+3. **Re-insertion Prevention**: What's the most efficient approach to prevent re-inserts of deleted keys?
+
+4. **Concurrency Model**: How do we best handle concurrent access to tree states during parallel validation?
 
 # Future possibilities
 [future-possibilities]: #future-possibilities
+
+1. **Pruning Strategies**: Implement advanced pruning strategies for historical tree states to reduce storage requirements.
+
+2. **Parallelization**: Further optimize tree state updates for parallel processing.
+
+3. **Checkpoint Verification**: Use tree states for efficient checkpoint-based verification.
+
+4. **Light Client Support**: Leverage tree states to support Zebra-based light clients with efficient proof verification.
+
+5. **State Storage Optimization**: Investigate more efficient serialization formats and storage mechanisms for tree states.