Skipped/Invalid tx for parachain with lookahead collator

[Kailai-Wang]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

Sorry that I opened this GH issue with my SE issue still pending: https://substrate.stackexchange.com/questions/12108/invalid-transaction-on-parachain-with-lookahead-collator

but this issue is quite important for us as we rely on these extrinsics from off-chain workers in our chain abstraction service.

I did some experiments locally and realised that it could be reproduced with standard template, steps:

1. build polkadot-sdk-parachain-template from the repo, revision is paritytech/polkadot-sdk-parachain-template@ecaf71d as of my case
2. download polkadot binary via zombienet setup polkadot -y
3. launch network with 2 relay chain nodes + 1 parachain node based on the following zombienet config:

[settings]
timeout = 1000
provider = "native"

[relaychain]
default_command = "polkadot"
chain = "rococo-local"

[[relaychain.nodes]]
name = "alice"
validator = true

[[relaychain.nodes]]
name = "bob"
validator = true

[[parachains]]
id = 1000
cumulus_based = true
add_to_genesis = true
register_para = true

[parachains.collator]
name = "Alice"
ws_port = 9944
command = "parachain-template-node"
args = [
  "--no-hardware-benchmarks",
  "--force-authoring",
  "--state-pruning=archive",
  "-l=parachain=debug,txpool=trace"
]

4. write a small script that repeatedly constructs and submits an extrinsic that transfers some balance from Alice to Bob:

import { ApiPromise, WsProvider, Keyring } from '@polkadot/api';
import { cryptoWaitReady } from '@polkadot/util-crypto';
import BN from 'bn.js'; // Ensure BN is imported for nonce manipulation

// Constants
const NODE_URL = 'ws://127.0.0.1:9944'; // Change if needed
const AMOUNT = 0.001 * 10 ** 18; // Transfer amount in smallest units (e.g., Planck)
const INTERVAL = 3000; // 3 seconds
const SENDER = '//Alice';
const RECEIVER = '//Bob';

const main = async () => {
  await cryptoWaitReady();

  // Initialize API and Keyring
  const provider = new WsProvider(NODE_URL);
  const api = await ApiPromise.create({ provider });
  const keyring = new Keyring({ type: 'sr25519' });
  const sender = keyring.addFromUri(SENDER);
  const receiver = keyring.addFromUri(RECEIVER);

  console.log(`Sender Address: ${sender.address}`);
  console.log(`Receiver Address: ${receiver.address}`);

  // Fetch initial nonce for the sender
  let nonce = new BN((await api.rpc.system.accountNextIndex(sender.address)).toString());
  console.log(`Starting Nonce: ${nonce.toString()}`);

  setInterval(async () => {
    try {
      console.log(`Submitting transaction with nonce ${nonce}...`);
      const transfer = api.tx.balances.transferKeepAlive(receiver.address, AMOUNT);

      // Sign and send the transaction
      const unsub = await transfer.signAndSend(sender, { nonce }, (result) => {
        if (result.status.isInBlock) {
          console.log(`Transaction included in block: ${result.status.asInBlock}`);
        } else if (result.status.isInvalid) {
          console.error('Transaction is invalid');
        } else if (result.status.isUsurped) {
          console.error('Transaction is usurped by another with the same nonce');
        } else if (result.status.isDropped) {
          console.error('Transaction is dropped from the pool');
        } else if (result.status.isFuture) {
          console.error('Transaction is in the future');
        } else if (result.status.isBroadcast) {
          console.log('Transaction broadcasted to peers');
        }
      });

      nonce = nonce.add(new BN(1));
      unsub();
    } catch (err) {
      console.error('Error submitting transaction:', err);
    }
  }, INTERVAL);
};

main().catch(console.error);

Observation:

• extrinsics are processed normally in the first few blocks
• starting from some point, no extrinsic is processed (there's no event) - occassionally there's a block that executes multiple transfer extrinsics as if they were processed in batch
• after a while, submitting extrinsics starts to show Future error:

Sender Address: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY
Receiver Address: 5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty
Starting Nonce: 0
Submitting transaction with nonce 0...
Submitting transaction with nonce 1...
Submitting transaction with nonce 2...
Submitting transaction with nonce 3...
Submitting transaction with nonce 4...
Submitting transaction with nonce 5...
Submitting transaction with nonce 6...
Submitting transaction with nonce 7...
Submitting transaction with nonce 8...
Transaction is in the future
Submitting transaction with nonce 9...
Transaction is in the future
Submitting transaction with nonce 10...
Transaction is in the future
Submitting transaction with nonce 11...
Transaction is in the future

I had the impression (sorry I couldn't find the exact GH issue) that running rococo-local as relay chain locally could cause frequent re-org, but my questions are:

• is it possible to mitigate this even with reorg? From client's perspective, it is doing nothing wrong - it constructs the extrinsic with incremented nonce and submits it, upon submission it's also accepted by the node (before the future error), but somehow these extrinsics get silently ignored/invalidated. This has a big impact on our service :(
• this problem also happens on our paseo parachain, where I thought reorg would occur less frequently but in fact not. Is this normal? It is concerning for us to upgrade our node/runtime in the main-net

Thanks for any help!

I attach the full collator log here in the repro-case too, on paseo we see quite some tx reported as invalid, the example logs were put in the linked SE issue above.

Alice.log

Steps to reproduce

No response

[bkchr]

@Kailai-Wang this should be: #1202

[Kailai-Wang]

@Kailai-Wang this should be: #1202

Thank you! I read through this thread - for now we'd just live with the workaround.

We'll wait until the stable release that includes the fa txpool.

I'll close this issue, thanks again for the help