[ProjectTracking]: Stateless validation StatelessNet

[walnut-the-cat]

Goals

• Demonstrate stateless validation to public so they have better understanding on what we are working on
• Leverage community power to identify critical issues and stress test the network with stateless validation
• Figure out detailed MainNet launch requirement and plan based on findings

Links to external documentations and discussions

• StatelessNet planning doc: LINK
• NEP: LINK

Estimated effort

• StatelessNet launch preparation: 2-3 weeks
• StatelessNet duration: 8+ weeks
• StatelessNet data analysis/post-mortem: 2 weeks
• [Out of scope] Implementation of the follow up items identified during StatelessNet

Timeline

Start StatelessNet stage 1: Late January/Early February

We aim to open StatelessNet to the public on January 29th so community members can test features and generate traffic.

Requirements

• 3 weeks before
  • StatelessNet plan public announcement should be made for visibility via multiple channels including X, Telegram, and Near.org.
• 2 weeks before
  • Stateless validation MVP should go through basic tests (unit tests, integration tests, load tests).
• 1 week before
  • Stateless validation MVP must be merged into master and be ready for distribution.
  • StatelessNet network should be set up along with necessary monitoring tools.

EthDenver: Feb 29th - March 3rd, 2024

Bowen Wang plans to share findings from Stateless validation StatelessNet during EthDenver.

Requirements

• 1 week before
  • Data from StatelessNet should be gathered and analyzed in presentable format.
• 2 weeks before
  • At least 1-2 rounds of performance optimization and stability enhancement should be in place based on incoming data.

Start StatelessNet Stage 2: Early March (at least 4 weeks after stage 1)

Expand scenario coverage of StatelessNet to start testing corner cases.

Mid-point check-in: March 15th, 2024

Continue collecting data and make improvements.

Complete StatelessNet: March 31st, 2024 (or TBD)

• Wrap up the StatelessNet test and share the final report.
• Need to have enough confidence on resilience, scalability, reliability of the network before completing StatelessNet and the end date may move around.

Assumptions

• NF will support public communication amplification
• Project is approved by NF with enough budgets
• At least 20 validators will be willing to participate to test stateless validation before MainNet launch
• Public interest will be high enough to run meaningful reward program

Pre-requisites

• MVP is complete and StatelessNet binary is ready by end of January
• StatelessNet set up and configuration are ready by end of January

Out of scope

• TestNet/MainNet launch
• Implementation of items identified during StatelessNet
• StatelessNet generalization for future use

TODO

Before StatelessNet launch - Core

• Complete MVP
• E2E integration test
• Internal sanity test

Before StatelessNet launch - Node

• NearOne operated nodes set up
  • Validator nodes (25 operated by NearOne) (Andrei) #24
    • Note: Don't need the entire 25 from the beginning. @robin-near suggests 3 validator nodes, 3
      chunk proposers (and we run 3 shards) at the beginning.
  • Archival nodes (2 nodes, 1 split-storage, 1 legacy) (Andrei) #26
  • RPC nodes (6 nodes: 2 per region) (Andrei) #25
  • State dumping node (1 node) (Andrei) #27
  • Backup generating nodes (1 node) (Andrei) #28
• User support
  • Wallet
  • Explorer backend (Bohdan + Andrei) #29
  • NEAR-CLI support (Marcelo) #30
  • Explorer frontend (Bohdan + Andrei) #37
  • Plan for how StatelessNet issues are triaged and addressed #44
  • NEARUP support
• StatelessNet framework support
  • Traffic generator (Marcelo) #41
  • Staking Pool Factory + Faucet (Marcelo) #33
  • Create a mirror of mainnet, and change account keys. Create the Statelessnet genesis. (Marcelo) #34
  • StatelessNet binary prep strategy for frequent release
  • Node update bash script (Andrei + Marcelo) #38
  • Easy configuration change support
    • Configure Short 3h epochs (Marcelo) #35
    • Increase per chunk gas limit to a larger number to verify the projected performance improvements (Marcelo) #36
    • Skip timeout
    • Validator role
    • Hardfork (for stateful -> stateless migration test)
  • Initial StatelessNet environment configuration
    • Automated creation of user accounts
    • Automated deployment of several smart contracts for testing purpose
• Metrics
  • Node uptime counter (Andrei + Marcelo) #40
  • Telemetry collection from nodes (Andrei+Bohdan) #39
  • monitoring dashboard
  • Debug page
• 🔷 [ProjectTracking] Enable mainnet validators to track a single shard nearcore#10462

Before StatelessNet launch - Operational

• Finalize
  • Public communication message
  • Reward program
  • High level timeline
• Get budget alignment with NEAR Foundation
• Create
  • An official support group on Telegram (Olga) #32
    • Potential name:
      • Stake Wars IV general support
      • Stake Wars IV validator support
    • Should promote channel in the following telegram channels:
      • NEAR Dev
      • NEAR Protocol community
      • NEAR Tools Community
      • Protocol validator channel (Do not know the exact name...)
  • Create Zulip Channel
    • pagoda/core/stake_wars_iv
  • GitHub repository for reward program
    • Note: a single Github repo maybe used for both program description as well as reward program
    • Potential name: near/stakewars_iv (similar to https://github.com/near/stakewars-iii)
    • Necessary contents
      • Program overview
      • Reward program overview
      • Issue template / label for submission
      • Reference links
• Get X account access (or figure out a way to make a post on NEAR X account)
  • account to use: @NearStakeWars
• Locate proper communication channel on Near.org
  • Landing page that presents a summary of the program + statistics about how the network is performing (Olga) #31
  • account to use: stake-wars.near
• Partner communication
  • Aurora
  • SWEAT
  • Kaiching
• Document to write
  • Reward program repo README
  • Stateless validation overview
  • NEP draft
• Communicate with validators on StatelessNet

Before StatelessNet Stage 2

TBD

Operational

• Google form for validator application
  • https://forms.gle/Nmdq6YAgTKD4pMqGA
  • What are the information we should gather?
    • MainNet account name
    • Node resource set up that will be used for validator node
    • Region
    • agreement to enable telemetry/profiling/debug page
    • agreement for automatic binary update?

Techincal

• Stress test / load test
• Performance test: Due. Feb 21st
  • We would like to have some performance data on what stateless validation and In-memory trie can enable. For instance, performance improvement data for fungible token transfer per sec per shard. We acknowledge that the number may not be accurate until all optimization/stabilization/bug-fixes are in, but Bowen can use such data to give high level overview during EthDenver.

After StatelessNet completion

TBD

[Longarithm]

Feb 8th Core team update

• Set up first branch, node configs and simple release process for statelessnet
• Solving issues
  • Hacky way to retry applying orphan state witness if the parent block doesn't yet exist nearcore#10535
  • Too many missing chunks on StatelessNet start nearcore#10560
• Started benchmarking
• Monitoring tools
  • Technical dashboard https://nearinc.grafana.net/d/edbdp75fzrklcb/stateless-validation?orgId=1&var-chain_id=statelessnet
  • Debug UI Add EntityDebug UI entities for stateless validation nearcore#10524

[walnut-the-cat]

Feb 26th update

• Majority of issues discovered at the beginning of stake wars program have been resolved now
• Orphan state pool implementation is now merged. The team will re-enable mainnet traffic mirroring soon
• [WIP] Testloop refactoring is in progress; the work is necessary to enable in-memory trie with proper testing
• Proper genesis handling & test fixes
• [WIP] Performance refactoring iteration based on Bowen's latest feedback. Plan to share the result by Feb 27th.
• The team started reviewing incoming reward program proposals to prepare for start of the program with Stage 2
• Community validator onboarding in progress
• Other MainNet release related topics will now be tracked in the tracking issue 46

[bowenwang1996]

[WIP] Testloop refactoring is in progress; the work is necessary to enable in-memory trie with proper testing

How much time is this going to take?

Proper genesis handling & test fixes

What does this mean?

[walnut-the-cat]

Regarding testloop, it's being discussed in this Zulip thread. I expect it to be mostly done this week, but am double checking with Robin.

[walnut-the-cat]

Proper genesis handling & test fixes

This is part of @pugachAG 's Better Engineering effort to make code base more manageable and easy to develop. So far, these two PRs are related:

• fix: properly handle genesis as part of stateless validation nearcore#10633
• [stateless_validation] Handle production of state witness for first chunk after genesis nearcore#10502

It will be followed by removal of mock epoch manager

[wacban]

March 11th Update

• discovered and debugging a number of issues
  • stack overflow - almost fixed
  • one shard stalling under mainnet load - under investigation
  • crash in memtrie - under investigation (cc. @staffik)
  • empty blocks and missing chunks - looking for an owner
• the next milestone, in-memory-trie and state sync integration - implementation is in progressder

[wacban]

April 5th Update

We deployed a number of new features to statelessnet but unfortunately the chain suffered tremendously due to bugs. We decided to shut down the chain as maintaining it on unstable code was consuming too much developers' time. We are going to continue development and testing on dedicated test networks. We may resume statelessnet once we've gained more confidence in it.

• finished features
  • single shard tracking using state sync
  • memtrie loading and unloading during state sync
  • soft state witness size limit
• fixed bugs
  • missing trie value issue
  • missing storage recording issue
  • (statelessnet only) reward calculation panic due to arithmetic overflows
  • (statelessnet only) maintenance scripts accidentaly unstaking all tokens
• misc
  • Attempted mirroring mainnet traffic but failed due to networking constraints. Search for a solution is in progress.

[walnut-the-cat]

Moving the task to 'Ready to be prioritized' from 'WIP' as we are pausing StatelessNet