2025 01 14 Testnet Rollback and Restart

Edit

As of 2025-01-16 13:00 UTC testnet is back online. The instructions below are no longer relevant. Nodes that haven't yet joined the cluster will need to update their shred version, download a snapshot, and start normally:

--expected-shred-version 64475 \

---

This testnet restart is NOT urgent. Follow these instructions when you have time, but don’t skip sleep or disrupt other plans for this.

Summary

In order to restart testnet on v2.0.22 we need to roll back 8 feature gates that are not yet enabled on mainnet-beta.

Attribute	Value
Validator version	Agave: v2.0.22 (and v2.1.9 for creating the snapshot)
Snapshot slot	312256120
Restart slot	312256121
Shred version	64475
Expected bank hash	6A7U1X8WqnWRXWtqaxF15sSYHLT66j1ycBEyFigGwr2Z

Step 1. Stop validator process if you haven’t already

Step 2: Install Agave v2.1.9

This is necessary in order to create the correct snapshot in step 3.

Agave: agave-install init v2.1.9

Frankendancer: Please switch to Agave for this restart and the subsequent upgrade/downgrade cycle.

Step 3. Create snapshot

This command creates a snapshot but removes 1 activated v2.1 feature gate account.

agave-ledger-tool --ledger <ledger-path> create-snapshot \
--incremental \
--snapshot-archive-path  <snapshot-path> \
--hard-fork 312256120 \
--deactivate-feature-gate \
    3opE3EzAKnUftUDURkzMgwpNgimBAypW1mNDYH4x4Zg7 \
    CLCoTADvV64PSrnR6QXty6Fwrt9Xc6EdxSJE4wLRePjq \
    tSynMCspg4xFiCj1v3TDb4c7crMR5tSBhLz4sF7rrNA \
    4eohviozzEeivk1y9UbrnekbAFMDQyJz5JjA9Y6gyvky \
    2Fr57nzzkLYXW695UdDxDeR5fhnZWSttZeZYemrnpGFV \
    8U4skmMVnF6k2kMvrWbQuRUT3qQSiTYpSjqmhmgfthZu \
    CGB2jM8pwZkeeiXQ66kBMyBR6Np61mggL7XUsmLjVcrw \
    CJzY83ggJHqPGDq8VisV3U91jDJLuEaALZooBrXtnnLU \
--enable-capitalization-change \
--  312256120 <snapshot-path>

The output should include this at (or near) the end:

    Successfully created snapshot for slot 312256121, hash 6A7U1X8WqnWRXWtqaxF15sSYHLT66j1ycBEyFigGwr2Z, base slot: <BASE_SLOT>>: /home/sol/ledger-snapshots/incremental-snapshot-<BASE_SLOT>-312256121-<SNAPSHOT_HASH>.tar.zst
    Capitalization change: -7628160 lamports
    Shred version: 64475

The capitalization change is expected because we deactivated a feature gate.

Note that each operator's snapshot file name may contain different base slot number and hash, but

• the bank hash should be 6A7U1X8WqnWRXWtqaxF15sSYHLT66j1ycBEyFigGwr2Z
• the second slot number should be 312256121
• the shred version should be 64475

Once you have created a snapshot move all the other snapshots to a backup directory, so your snapshot directory contains one full snapshot and one incremental snapshot. Note that the <BASE_SLOT> in these two filenames should match.

snapshot-<BASE_SLOT>-<BASE_SNAPSHOT_HASH>.tar.zst
incremental-snapshot-<BASE_SLOT>-312256121-<SNAPSHOT_HASH>.tar.zst

If you fail to create a snapshot see the appendix for possible fixes.

Step 4: Install Agave v2.0.22

This is the version we want to use to restart the cluster.

Agave: agave-install init v2.0.22

Step 5: Update startup config and start your validator

Agave

Add these arguments to your validator startup script:

--wait-for-supermajority 312256121 \
--expected-shred-version 64475 \
--expected-bank-hash 6A7U1X8WqnWRXWtqaxF15sSYHLT66j1ycBEyFigGwr2Z \

As it starts, the validator will load the snapshot for slot 312256121 and wait for 80% of the stake to come online before producing/validating new blocks.

To confirm your restarted validator is correctly waiting for 80% stake, look for this periodic log message to confirm it is waiting:

INFO  solana_core::validator] Waiting for 80% of activated stake at slot 312256121 to be in gossip...

And if you have RPC enabled, ask it for the current slot:

solana --url http://127.0.0.1:8899 slot

Any number other than 312256121 means you did not complete the steps correctly.

Once started you should see log entries for “active stake” visible in gossip and “waiting for 80% of stake” to be visible. You can track these to see how the stake progresses.

---

Appendix (use this only if step 3 failed)

If you get an error like this:

Error: Slot 312256120 is not available

Or this:

Unable to process blockstore from starting slot <slot> to 312256120; the ending slot is less than the starting slot. The starting slot will be the latest snapshot slot, or genesis if the --no-snapshot flag is specified or if no snapshots are found.

Your snapshots directory contains a snapshot that is for a slot >312256120. If you also have a snapshot for slot <=312256120 then move snapshots for slots >312256120 to a backup directory and run the agave-ledger-tool command again. If you do not have a snapshot for slot <=312256120 then you will need to download a snapshot

If you successfully created a snapshot, resume the instructions above starting at Step 4. If you are unable to create a snapshot, follow the instructions below on downloading a snapshot.

If you couldn’t produce your snapshot locally follow these appendix steps

Step 1: Download a snapshot from a known validator

If you are unable to generate a snapshot locally for slot 312256121 you will need to download one from a known validator. Add these lines to your startup script.

--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
--expected-shred-version 64475 \

Remove the flag --no-snapshot-fetch in your startup script if it is present.

Step 2: After download, restart

Verify that you have a new snapshot in your snapshot directory. If the snapshot is done downloading, stop your validator process.

Add the flag --no-snapshot-fetch to your startup script

Resume the instructions above starting at Step 4.