[Feature Request]: allow user to set account sequence

[jcstein]

Implementation ideas

Users (including myself) have reported that they are hitting account sequence errors in posting blobs using celestia-node. The workaround for this is to set the correct account sequence. While it is possible to set the account sequence in celestia-app's binary, it isn't possible with celestia-node.

This feature's implementation would allow the user to send a transaction using celestia-node with a custom account sequence.

[distractedm1nd]

We could maybe do this for state.SubmitPayForBlob but definitely not blob module. This functionality is technically already supported via SubmitTx no?

[jcstein]

the only param I see is the data on state.SubmitTx?

[jcstein]

I don't think this is possible either on other methods, based on some testing I'm doing with the CLI at the moment.

celestia state delegate celestiavaloper1ukk0l6ce58zq3snad5vn24vy7pdk9r8hva6nz9 1 10000 100000 --node.store $NODE_STORE
{
  "result": "rpc error: code = Unknown desc = timed out waiting for tx to be included in a block"
}

celestia state delegate celestiavaloper1ukk0l6ce58zq3snad5vn24vy7pdk9r8hva6nz9 1000 10000 100000 --node.store $NODE_STORE
{
  "result": {
    "txhash": "497CA6B400C9040A3771D4D63A64EAD2C6DCCD481975D4719A33F49EA53A1676",
    "codespace": "sdk",
    "code": 32,
    "raw_log": "account sequence mismatch, expected 624, got 623: incorrect account sequence",
    "logs": null,
    "gas_wanted": 100000,
    "gas_used": 46383,
    "events": null
  }
}

celestia state delegate celestiavaloper1ukk0l6ce58zq3snad5vn24vy7pdk9r8hva6nz9 1000 10000 100000 --node.store $NODE_STORE --help
Sends a user's liquid tokens to a validator for delegation.

Usage:
  celestia state delegate [valAddress] [amount] [fee] [gasLimit] [flags]

Flags:
  -h, --help   help for delegate

Global Flags:
      --node.store string   The path to root/home directory of your Celestia Node Store
      --token string        Authorization token
      --url string          Request URL (default "http://localhost:26658")

[jcstein]

How would a user submit multiple blobs in the same block with blob.Submit or state.SubmitPayForBlobs?

[mindstyle85]

this would be really good to have, since we already saw one case where his actual vs expected sequence was not the same (due to cosmos ofc, its known this can happen), but since he was using just the da node for that, he had no way to reset it (would either require a restart of the rpc he used or import to app and use the --sequence flag)

[jcstein]

yeah, i also want to make sure the documentation that says it is possible, is actually possible. at version v0.13.0 i cannot figure out how to set account sequence through celestia-node

[bkolad]

My observation:

While submitting blobs one after another using the blob_submit(&self, blobs: &[Blob], opts: SubmitOptions) -> Result<u64, Error>, I encountered errors related to incorrect account nonce. It appears that there might be a race condition between requests. I initially assumed that the synchronization would be managed internally since the method takes &self. After wrapping the client with a Mutex, the issue disappeared, but it might be matter of timing.

[Wondertan]

The account sequence(nounce) is not managed smart enough in the node and this is the core issue here.

We could enable account sequence control for our users, but we would then outsource the complexity away, forcing every user to manage nounce on their own. Instead, we should abstract the complexity and simplify life for our users.

cc @cmwaters

[jcstein]

so how does one do this either on celestia-node or celestia-app? that's the thing i cannot figure out

[Wondertan]

Users (including myself) have reported that they are hitting account sequence errors in posting blobs using celestia-node. The workaround for this is to set the correct account sequence.

The root issue here is the lack of smarts around nounces. Once we have them, your issue disappears, and setting the account sequence won't be necessary on the node side.

so how does one do this either on celestia-node or celestia-app?

So, one wouldn't need to do it on the celestia-node side in the first place.

[jcstein]

will it allow someone to submit multiple transactions from the same account in one block? like described in this section

[rootulp]

Relevant: celestiaorg/celestia-app#3196 which improved nonce handling for the celestia-app signer (in the user package). AFAICT celestia-node doesn't use the celestia-app user package yet but celestia-node is considering migrating to it (see #2650).

[Wondertan]

will it allow someone to submit multiple transactions from the same account in one block?

Yes in case both transactions turn to be valid. If one of them is invalid, things might be a bit more complicated.

[vgonkivs]

I can assume that this issue will not appear after signer integration as it's using local sequence number.
@rootulp, @cmwaters can you please confirm this.

[rootulp]

cc: @ninabarbakadze who's working on signer stuff while @cmwaters is OOO

[cmwaters]

Yes this should solve the issue of submitting multiple blobs in parallel (so long as the user is not using multiple instances of the signer)

[vgonkivs]

Hey @jcstein, since the issue has been already fixed, should celestia-node still support account sequencing?

[distractedm1nd]

Hmmmm. I think it should still be an option. But it is definitely a lot lower priority now. It can be shipped together with the rest of the blob submit options, and TxOptions in state mod

[jcstein]

thanks @distractedm1nd and @vgonkivs. let's go with distractedm1nd's suggestion

[jcstein]

yeah, while trying to use the golang tutorial as a template, when I run main.go I hit tx already in mempool. how do I work around this? if I increase gas price, I get account sequence error

$ go run main.go
2024/05/23 19:09:23 Error submitting blob: : tx already in mempool
exit status 1

$ celestia blob submit 0x4269 0x4269 --node.store ~/.celestia-light --gas.price 0.01
{
  "result": ": incorrect account sequence"
}

[jcstein]

note: this was on mainnet so the app version may not be the one with the fix included?

[cmwaters]

@jcstein if the tx is already in mempool then what's actually the problem?

For me, I think nonce's should be an internal detail that isn't exposed to users so ideally we smooth out these bugs rather than passing that baton to users to handle

[liamsi]

I think Josh is just trying to replicate this. Apparently, RaaS providers want to prep and submit several txs and set their nonces accordingly (instead of preparing one after the other, s.t. the nonces get auto-updated from state). Is this correct @Bidon15 @jcstein, or am Im misunderstanding this?

[Bidon15]

Apparently, RaaS providers want to prep and submit several txs and set their nonces accordingly (instead of preparing one after the other, s.t. the nonces get auto-updated from state). Is this correct @Bidon15 @jcstein, or am Im misunderstanding this?

Yes. This is born from the issue when they need to resubmit the blob, bc of the failed previous one. If we can do it automatically, then it's awesome.

If we can't do automatically, then at least giving them self-control is needed. Otherwise, they won't be able to resubmit blobs and be stuck in incorrect sequence number

Here is the gist of what our RaaS partners are experiencing

WARN [06-05|23:00:06.822] Blob Submission error                    err="rpc error: code = Unknown desc = failed to subscribe to tx: already subscribed"
WARN [06-05|23:00:06.823] Falling back to storing data on DAC      err="rpc error: code = Unknown desc = failed to subscribe to tx: already subscribed"
INFO [06-05|23:00:07.035] DataPoster sent transaction              nonce=5  hash=046fc7..12ac51 feeCap=51,055,105,760 tipCap=99,078,945 blobFeeCap=<nil> gas=1,538,064
INFO [06-05|23:00:07.038] BatchPoster: batch sent                  sequenceNumber=6 from=22 to=23 prevDelayed=8 currentDelayed=8 totalSegments=3 numBlobs=0
WARN [06-05|23:00:07.346] Blob Submission error                    err=": incorrect account sequence"
WARN [06-05|23:00:07.346] Falling back to storing data on DAC      err=": incorrect account sequence"
INFO [06-05|23:00:07.559] DataPoster sent transaction              nonce=6  hash=261475..5c08bc feeCap=51,055,105,760 tipCap=99,078,945 blobFeeCap=<nil> gas=1,540,640
INFO [06-05|23:00:07.562] BatchPoster: batch sent                  sequenceNumber=7 from=23 to=24 prevDelayed=8 currentDelayed=8 totalSegments=3 numBlobs=0
WARN [06-05|23:00:07.816] Blob Submission error                    err=": incorrect account sequence"
WARN [06-05|23:00:07.816] Falling back to storing data on DAC      err=": incorrect account sequence"
INFO [06-05|23:00:08.020] DataPoster sent transaction              nonce=7  hash=ebcae6..1afe5a feeCap=51,055,105,760 tipCap=99,078,945 blobFeeCap=<nil> gas=1,542,509

[distractedm1nd]

Node will not add the option to set the account sequence until the new signer/client library from @cmwaters is tried.

Callum made a very good point internally that "Users shouldn’t need to know that nonces exist. Giving them control is just a cop out because we can’t design a good system ourselves.". If it remains an issue we can add manual nonce control.

[distractedm1nd]

@jcstein 's case is not an example of this issue - it is the exact same transaction submitted twice, after receiving a "tx already in mempool error"

[jcstein]

we can close this PR and reopen it if it happens again? AFAIU it is fixed in v0.13.7 celestia-node release

How would a user submit multiple blobs in the same block with blob.Submit or state.SubmitPayForBlobs?

however, I think it would be good to separately revisit this section in our docs. if it is only possible in celestia-app, we should make that clear. and make it clear it is not possible to set your own account sequence in celestia-node

[jcstein]

@jcstein if the tx is already in mempool then what's actually the problem?

it gets stuck in the mempool. unreliable tx submission. the issue is that there's no way to (1) increase gas (2) increase nonce, and resubmit

For me, I think nonce's should be an internal detail that isn't exposed to users so ideally we smooth out these bugs rather than passing that baton to users to handle

Although it is a detail, we do mention this is possible in the Celestia docs

@jcstein 's case is not an example of this issue - it is the exact same transaction submitted twice, after receiving a "tx already in mempool error"

Although it is the same transaction, IMO increasing gas and nonce should allow it to be forced to be included in the next block or two, assuming the fee market cooperates

[zvolin]

I can still reproduce this. In lumina for a long time we had a lock in rpc tests that sequenced calls to blob.Submit. I tried removing it and it still quite reliably results in:

---- blob_submit_and_get stdout ----
thread 'blob_submit_and_get' panicked at rpc/tests/blob.rs:24:72:
called `Result::unwrap()` on an `Err` value: Call(ErrorObject { code: ServerError(1), message: "broadcast tx error: account sequence mismatch, expected 5, got 6: incorrect account sequence", data: None })

For reproduction

git clone [email protected]:eigerco/lumina && cd lumina
docker compose -f ci/docker-compose.yml up --build --force-recreate -d
./tools/gen_auth_tokens.sh

# remove the mentioned lock
patch -p1 <<"EOF"
diff --git a/rpc/tests/utils/client.rs b/rpc/tests/utils/client.rs
index 55beb7d..1e7681c 100644
--- a/rpc/tests/utils/client.rs
+++ b/rpc/tests/utils/client.rs
@@ -55,6 +55,6 @@ pub async fn blob_submit<C>(client: &C, blobs: &[Blob]) -> Result<u64, ClientErr
 where
     C: SubscriptionClientT + Sync,
 {
-    let _guard = write_lock().await;
+    // let _guard = write_lock().await;
     client.blob_submit(blobs, TxConfig::default()).await
 }
EOF

# run rpc tests
cargo test -p celestia-rpc

After it reproduces, all subsequent calls fail with this error

[Wondertan]

@zvolin, it's likely related to #3980