author_submitAndWatchExtrinsic reports incorrect block hash when there are missed authorship slots

[kylezs]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

What we've seen on the Chainflip network:

1. Submit an extrinsic with author_submitAndWatchExtrinsic and eventually get InBlock(A)
2. The finalised stream says the extrinsic was in the block before block A... however the block hash A still exists.

I'm assuming it has to do with the fact there were missed authorship slots at that time. In block A (on the finalised chain) we see that there are two missing authorship events, meaning that the authors of the two previous blocks missed their slots.

This behaviour from the subscription, would still be incorrect though, since even in the case of a reorg, the fact the extrinsic was in a different block would change the hash of block A, but the block with hash A still exists, without the extrinsic we were looking for.

We are using Aura for block authorship.

Steps to reproduce

No response

[bkchr]

So, what you are saying it reports InBlock(A), but the extrinsic is not in block A?

CC @michalkucharczyk

[michalkucharczyk]

Is it fork-aware pool or the old one? (Did you pass --pool-type=fork-aware?)

[michalkucharczyk]

Would you also provide a version of the node used?
What is network configuration: do you send transaction directly to collator, or does it go via dedicated RPC node?

[kylezs]

Thanks for taking a look into this.

So, what you are saying it reports InBlock(A), but the extrinsic is not in block A?

Yes, that's correct.

Would you also provide a version of the node used?

We using polkadot-sdk version 1.15 - in particular this fork: https://github.com/chainflip-io/polkadot-sdk/tree/chainflip-substrate-1.15.2%2B2
There are only very minor changes to this fork, I think the only change is making the event types public, can't recall from memory exactly, but certainly nothing to do with these subscriptions, should be the same as 1.15 in that respect.

do you send transaction directly to collator?

In this case, they are sent directly to one of the authoring nodes via localhost. There's effectively a side-car component reading from the validator/author that is listening and submitting extrinsics to it, though ofc, this node isn't always the author.

Is it fork-aware pool or the old one?

It's not fork-aware. I wasn't aware of fork-aware 😄

[michalkucharczyk]

What is the reproducibility rate?

[kylezs]

I've only seen it once unfortunately. Since the network has had less congestion since, there have been fewer missed authorship slots and so there is already only a small chance that we would even get into a situation it was possible for this to occur.

[kylezs]

Just hit it again. Exactly the same conditions.

• Expected it to be found in block X according to author_submitAndWatchExtrinsic
• Ends up in the block before X on finalised chain.
• Block with hash X exists, and has a missed authorship slot penalty event, so someone missed authorship in the block before X.

[michalkucharczyk]

Seems like missing an authorship slot is an important puzzle here.

@bkchr is it something that can be easily emulated in zombienet/local network? I can run some tests, maybe I would be able to reproduce it. Maybe it is also affecting new pool - would like to cross check this.

[bkchr]

@bkchr is it something that can be easily emulated in zombienet/local network? I can run some tests, maybe I would be able to reproduce it. Maybe it is also affecting new pool - would like to cross check this.

Just stop the node and then it can not produce for a slot :D Or maybe do not even start the node.

[dandanlen]

Our suspicion is that this is somehow reorg-related, so I don't think stopping a node will work.

All nodes need to be able to author blocks but some of them should author for the wrong slot - like maybe tell some of them to author for slot S+1, or simulate clock drift by messing with timestamps.

I think it's something like this: the node that processes the submission reports that it's included in block A at height N, but that block is eventually reorged out of the canonical sequence and it just so happens that in the canonical sequence of blocks, the tx was already in block B at height N-1.

Block A would still be queryable by hash on the block explorer, but it's no longer part of the canonical chain.

Does the above sound possible/probable?

Maybe this can be solved by using the new transactionWatch apis?

[michalkucharczyk]

I am afraid that w/o quick path to reproduce the issue there is not much that I can do.

Would it be possible that you attach the log, when you hit the issue? I'd need txpool=trace level enabled. Please attach it here, with the hash of wrongly reported block, and hash of the related transaction.

[dandanlen]

@michalkucharczyk I know it's not a simple one... I'll see if I can get hold of some logs. Thanks for your help.

Can you point me towards some details/documentation about the new fork-aware pool? It sounds like the kind of thing that might help.

[michalkucharczyk]

Maybe this post could be a good start: https://forum.polkadot.network/t/the-fork-aware-transaction-pool/10468
(You can find further references there).