Epic: Blockchain Scanning

[mpguerra]

Motivation

We want zebra to be able to scan shielded transactions using viewing keys. This issue will track the steps and work necessary in order for zebra to implement blockchain scanning.

Scope

Feature: De-couple zebra-scan from zebrad

• The scanner will communicate with Zebra for the finalized part of the blockchain with a ReadStateService::init method: Add an init function for a standalone ReadStateService #8520
• We need a replacement for ChainTipChange so we can use it in zebra_scanner::spawn_init: Add a current_tip() or similar to ReadState #8580
• Create a new binary for the new process: Create a new binary zebra-scanner #8581
• Get rid of scanner startup code from zebrad: Remove scanner starting from zebrad #8582
• Move or remove the possibility of keys in the scanner: Move or remove the possibility of keys in the config file for scanning #8584
• Testing:
  • Testing the new zebra-scanner binary #8585,
  • change(scan): Move integration tests for scanner out of zebrad #8593
• Documentation: Documentation changes for the new zebra-scanner binary #8586

Non finalized state:

• Add a function for syncing blocks from Zebra's non-finalized best chain to a ReadStateService in another process #8576

Future Iterations

RPC Interface improvements

• Libraries for zebra-scan RPC Interface #8154
• Scanner RPC Design #8156
• Optional ScanService Requests
  • Implement Status scan service to get information related to a key #8226
  • Implement Auth scan service request for authorizing access to a set of keys and their scan results #8202
• Optional RPCs
  • feature: zebra-scan: Client side authentication #8164
  • Support JSON-encoded request/response data from gRPC server #8245
  • Extend getresults grpc #8233
• Refactor ScanTask in zebra-scan #8250
• Improve scans_for_new_key test #8236
• Refactor zebra-grpc and rename it to zebra-scan-rpc #8243

Key and scanning results database

• feat(zebra-db): Refactor format upgrades into a trait #7932
• feat(zebra-db): Split low level database code inside zebra-state #7926
• Create a zebra-db crate #7937
• Store decrypted scanning results in the scanner storage #7944
• Launch a task to clean up old scanner database versions
• Delete scanner results with empty transaction ID lists, except for the highest one for each key
• Delete results for keys that have been removed from the config (security)
• Store keys in the database with account IDs and birthday heights
• Reduce database size using small integer key IDs in the results table

Scanner

• feat(scan): Support batched scanning #8221
  • feat(scan): Associate scanning keys with AccountIDs #8229
• feat(zebra-scan): Immediately scan newly verified blocks #7905
• change(zebra-state): When the best chain changes, scan all the blocks in the best chain fork #7934
  • feat(zebra-state): Send a reference to latest non-finalized state in TipAction::Reset #7927
• feat(scan): Support various types of Sapling scanning keys #8231
• feat(scan): Support Orchard scanning keys #8232

Integration with zebrad

• Add a state request to get the block hash, height, and transaction index from the transaction ID #7943
• Compare config file and database after restart, act accordingly #8101

Security

• security: Files created by Zebra can be read by any user on the same machine #7807
• security: Run the scanner it is own tokio executor, and use low-priority threads to avoid timing attacks #8043
• Add a wrapper around private key types, so we never print them in display or debug code
• Store scanner viewing keys in a separate file, and give them more restrictive file permissions #8048
• Make the writing API of read-only databases inaccessible #8098
• Stop repeating the secret key in every database row, this makes it harder to securely delete from disk
• Store the private key in the database as a salted hash, so it can't be read and re-used
• Run the scanner in an isolated environment #8141

User Testing / Experimental Builds

• Warn if scan is configured but the scanning feature is not enabled, rather than panicking
• change(release): Enable all user features in the experimental build #7892

Performance Improvements

• Spawn one task per key or per 100 blocks, and limit the tasks to 1/4 of the available parallelism by default (this gives the verifiers some available physical cores, avoiding timing attacks)
• Only scan blocks that have sapling shielded outputs or shielded spends with sapling keys (combining the heights in sapling_note_commitment_tree and a new separate index for sapling spends): https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/state-db-upgrades.md#current-state-database-format
• Only scan blocks that have orchard shielded actions with orchard keys (using the heights in orchard_note_commitment_tree): https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/state-db-upgrades.md#current-state-database-format
• perf: Stop expensive cryptographic validation when deserializing shielded transactions #7939
• [ ] Batch multiple keys efficiently by scanning all keys, then repeatedly split the keys in half, and discard any unsuccessful keys, until we have all the keys that decrypt at least one note
• Scan recent blocks first: from the tip backwards in one task, and the tip forward in another
• Get block and transaction hashes from the state, rather than re-hashing the data
• Deserialize only the parts needed for compact blocks?

Broadcast channel

• feat(zebra-scan): Create a scanner results channel #7906
• feat(zebra-scan): Make the scanner send data to the results channel #7907

Implemented

Feature: RPC Interface - ✅

Issues

- [x] #8153 - [x] Implement `ScanService` Requests - [x] #8203 - [x] #8220 - [x] #8204 - [x] #8205 - [x] #8206 - [x] #8207 - [x] Implement RPCs - [x] #8161 - [x] #8162 - [x] #8163 - [x] #8176 - [x] #8235 - [x] #8242 - [x] #8244 - [x] #8256 - [x] #8285 - [x] #8286 - [x] #8259

PoC - ✅

Issues

- [x] #7727 - [x] #7903

MVP (v0.1) - ✅

For the MVP, let's assume the following:

• No external client (i.e. no RPCs)
• User has the necessary technical skills to query our database/datastore/however-we-store-the-scanning-results directly
• User controls their whole machine (so file permissions and secret data deletion don’t matter)
• We want to store the scanning results so that they can be made available to the User at a time of their choosing

Issues

Key and scanning results database

• feat(zebra-scan): Create a key storage database in RAM #7904
• feat(zebra-scan): Make the key storage database persistent #7928
• feat(config): Add keys to be scanned to Zebra config file  #7941
• Re-use existing raw database code for the scanner, without splitting the state yet #8001
• Implement scanning results storage format #8019
• Write scanning results to database #8021
• Change scanner database format so it is easier for users to query for transaction IDs #8050
• Add the RocksDB ldb tool to the Docker image, so technical users can query the database #8051
• feature: Consider allowing opening the database in "read-only" or "secondary" mode #8078

Scanner

• Create a scan_block function to use across scanning tasks #7947
• Scan keys one at a time, so we know which key(s) decrypt each transaction #8018
• Scan verified blocks from sapling activation to the tip #7953

Integration with zebrad

• feat(zebra-scan): Connect scanner with zebrad for block processing #7908
• feat(scanner): Read and store keys from config at zebra startup #7946
• feat(scanner): Start scanning where it left off last time #8022

Testing & Validation

• Run Zebra's unit tests with the scanner enabled #8037
• test: Add database snapshot tests for the scanner database #8069
• test(scan): Scan blocks containing ZECpages transactions with the ZECpages shielded key #7813

Documentation

• Document how to run and query the results of the blockchain scanning MVP #8046

Security

• Check if any of the security issues are blockers for the MVP, and document the remaining security issues #8049

Lower Priority Tests

• test: Add standard database tests to the blockchain scanner #8102
• Add scan_start_where_left test to CI #8085

References

• zcash_client_backend crate:

  • Code
  • Docs
  • Last Version: 0.10.0
  • scan_block: Returns a vector of transactions that matches the provided key, for the provided block.
    • Docs
    • Code
    • No orchard support: https://github.com/zcash/librustzcash/issues/403
  • scan_cached_blocks: Scan an amount of blocks from the Block Source:
    • Docs
    • Code

• zcash_client_sqllite crate:

  • Code
  • Docs
  • Last Version: 0.8.0
  • Crate description: zcash_client_sqlite contains complete SQLite-based implementations of the WalletRead, WalletWrite, and BlockSource traits from the zcash_client_backend crate. In combination with zcash_client_backend, it provides a full implementation of a SQLite-backed client for the Zcash network.
  • block source creation:
    • Code
    • Contains unstable features.

Related work

• Block Chain Scanning  #325
• https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/rfcs/0009-zebra-client.md

[mpguerra]

@oxarbitrage I've created a new section for the minimum viable product work and renamed the "Implementation" section to "Future iterations". Can you please move any issues that should be part of the MVP to this new section to help me with scheduling?

[arya2]

There's a discussion for the MVP here: #7975

[teor2345]

We aren't running the scanner task in CI yet, so I added this task to the MVP:

• Run Zebra's unit tests with the scanner enabled, by adding shielded-scan to the RUST_EXPERIMENTAL_FEATURES GitHub repository variable

zebra/.github/workflows/ci-unit-tests-docker.yml

Lines 133 to 135 in 1708f9d

	if [[ -n "${{ vars.RUST_EXPERIMENTAL_FEATURES }}" && "${{ vars.RUST_EXPERIMENTAL_FEATURES }}" != " " ]]; then
	docker run -e NETWORK --name zebrad-tests-experimental --tty ${{ vars.GAR_BASE }}/${{ vars.CI_IMAGE_NAME }}@${{ needs.build.outputs.image_digest }} cargo test --locked --release --features "${{ env.EXPERIMENTAL_FEATURES }} " --workspace -- --include-ignored
	fi

It takes less than a minute to enable it, but I wanted to check with the team first.

[mpguerra]

We've done all we're going to do here