[ZK Stack] - Sequencer crash on bridging the first ETH - priority transaction serial id mismatch

[moneta]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.84.1

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

I deployed a new zkChain on sepolia following the documents. The sequencer was up and running. The logs look normal as follows:

2025-03-06T22:33:12.401189Z  INFO zksync_node_api_server::web3: Successfully waited for at least one L1 batch in Postgres; the earliest one is #0
2025-03-06T22:33:12.448857Z  INFO zksync_node_api_server::web3: Initialized WS API on 0.0.0.0:3051
2025-03-06T22:33:12.448879Z DEBUG zksync_health_check: Changed health of `ws_api` from {"status":"not_ready"} to {"status":"ready"}
2025-03-06T22:33:12.464084Z  INFO zksync_metadata_calculator::updater: Next L1 batch for Merkle tree: 1, current Postgres L1 batch: L1BatchNumber(0), last L1 batch with metadata: Some(L1BatchNumber(0))
2025-03-06T22:33:12.464942Z  INFO zksync_metadata_calculator: Merkle tree is initialized and ready to process L1 batches: MerkleTreeInfo { mode: Full, root_hash: 0xe651f8df7b8bd54a803158def804382f08e11a7a5ab738b0121485193440690a, next_l1_batch_number: L1BatchNumber(1), min_l1_batch_number: Some(L1BatchNumber(0)), leaf_count: 67 }
2025-03-06T22:33:12.464965Z DEBUG zksync_health_check: Changed health of `tree` from {"status":"affected","details":{"stage":"initialization"}} to {"status":"ready","details":{"leaf_count":67,"min_l1_batch_number":0,"mode":"full","next_l1_batch_number":1,"root_hash":"0xe651f8df7b8bd54a803158def804382f08e11a7a5ab738b0121485193440690a","stage":"main_loop"}}
2025-03-06T22:33:12.464978Z  INFO zksync_metadata_calculator::updater: Initialized metadata calculator with 20 max L1 batches per iteration. Next L1 batch for Merkle tree: 1
2025-03-06T22:33:12.473527Z DEBUG zksync_health_check: Changed health of `state_keeper` from {"status":"not_ready"} to {"status":"ready","details":{"l1_batch":1,"next_l2_block":1,"prev_l2_block_hash":"0xe8e77626586f73b955364c7b4bbf0bb7f7685ebd40e852b164633a4acbd3244c","prev_l2_block_timestamp":0}}
2025-03-06T22:33:12.473544Z  INFO zksync_state_keeper::keeper: Starting state keeper. Next l1 batch to seal: 1, next L2 block to seal: 1
2025-03-06T22:33:12.473548Z  INFO zksync_state_keeper::keeper: There is no open pending batch, starting a new empty batch
2025-03-06T22:33:56.548622Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2025-03-06T22:34:11.565421Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2025-03-06T22:34:26.584586Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2

Then I tried to bridge 0.1Eth to L2. This is the successful tx hash on sepolia: https://sepolia.etherscan.io/tx/0x0516be6e6ead2354df9abbc22a61608fd54836e0d73b8a011745d78db11fcdab

But the sequencer crashes right after that with error: Error: One or more tasks failed: ["Task eth_watch panicked: assertion left == right failed: priority transaction serial id mismatch\n left: PriorityOpId(7)\n right: PriorityOpId(0)"]

Here are the log message:

ofGenerationDataRequest)
2025-03-06T22:46:12.531422Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2025-03-06T22:46:27.532791Z DEBUG zksync_web3_decl::client: Request to request `eth_blockNumber` by component `gas_adjuster` was rate-limited using policy 1 reqs/0ns: AcquireStats { was_waiting: true, total_sleep_time: 380ns } network="ethereum_11155111" component="gas_adjuster" origin=request `eth_blockNumber`
2025-03-06T22:46:27.556081Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2025-03-06T22:46:42.571207Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2025-03-06T22:46:57.586975Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2025-03-06T22:47:12.606500Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2025-03-06T22:47:27.628394Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2025-03-06T22:47:42.644902Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2025-03-06T22:47:57.663617Z  INFO get_proof_generation_data: zksync_proof_data_handler::request_processor: Received request for proof generation data: Json(ProofGenerationDataRequest)
2025-03-06T22:48:00.758716Z DEBUG EthWatch::loop_iteration: zksync_eth_watch::event_processors::priority_ops: Received priority requests with serial ids: 7 (block 7847857) - 7 (block 7847857)
thread 'tokio-runtime-worker' panicked at node/eth_watch/src/event_processors/priority_ops.rs:82:9:
assertion `left == right` failed: priority transaction serial id mismatch
  left: PriorityOpId(7)
 right: PriorityOpId(0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2025-03-06T22:48:00.761507Z ERROR zksync_node_framework::service: Task eth_watch panicked: assertion `left == right` failed: priority transaction serial id mismatch
  left: PriorityOpId(7)
 right: PriorityOpId(0)
2025-03-06T22:48:00.761523Z  INFO zksync_node_framework::service: One of the task has exited, shutting down the node

Could it be because of we reuse the wallets.yaml file when deploy a new chain? And so so transaction related to these wallets got pickup causing a mismatch? Or else, why are they mismatched?

Expected Behavior

The bridge transaction should be successful

Current Behavior

the bridge transaction failed and the sequencer crashes.

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[cytadela8]

This error indicates that some priority operations were not included when they should have been. Because "0" is on the right side, this means priority ops created during chain creation were not captured/detected by your server. You have previously modified the PRIORITY_EXPIRATION this might have affected those transactions being included by the server. By modifying the "processed_events" table you should be able to make the server refetch those.