Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: nonce_mismatch log parsing #4067

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

InoMurko
Copy link

@InoMurko InoMurko commented Nov 28, 2024

The actual error log that would bubble up is actually (using this in celestia-node)

broadcast tx error: account sequence mismatch, expected 357545, got 357544: incorrect account sequence

So seeking a prefix for account sequence mismatch would not work.

@InoMurko InoMurko requested a review from a team as a code owner November 28, 2024 13:16
@InoMurko InoMurko requested review from rootulp and evan-forbes and removed request for a team November 28, 2024 13:16
@celestia-bot celestia-bot requested a review from a team November 28, 2024 13:17
Copy link
Contributor

coderabbitai bot commented Nov 28, 2024

📝 Walkthrough

Walkthrough

The changes involve a modification to the ParseExpectedSequence function in the app/errors/nonce_mismatch.go file. The update replaces the use of strings.HasPrefix with strings.Contains to check for the error message substring "incorrect account sequence" instead of "account sequence mismatch." This adjustment alters how the function identifies specific error messages related to nonce mismatches while keeping the rest of the function's logic intact.

Changes

File Change Summary
app/errors/nonce_mismatch.go Modified ParseExpectedSequence to use strings.Contains for error message checking instead of strings.HasPrefix.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant System
    User->>System: Trigger nonce mismatch
    System->>System: Call ParseExpectedSequence
    System->>System: Check error message with strings.Contains
    alt Error message matches
        System-->>User: Return expected sequence number
    else Error message does not match
        System-->>User: Handle error
    end
Loading

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (1)
app/errors/nonce_mismatch.go (1)

Line range hint 35-45: Consider making the sequence number parsing more robust

The current implementation makes several assumptions that could make it fragile:

  1. It assumes the first number is always the expected sequence
  2. It discards the actual sequence number which could be useful for debugging
  3. The error message format might vary across different SDK versions

Consider enhancing the implementation to:

  1. Verify the numbers appear after "expected" and "got" keywords
  2. Return both expected and actual sequence numbers
  3. Add version-specific message handling
 func ParseExpectedSequence(str string) (uint64, error) {
 	if !strings.Contains(str, "incorrect account sequence") {
 		return 0, fmt.Errorf("unexpected wrong sequence error: %s", str)
 	}
 
 	numbers := regexpInt.FindAllString(str, -1)
 	if len(numbers) != 2 {
 		return 0, fmt.Errorf("expected two numbers in string, got %d", len(numbers))
 	}
 
-	// the first number is the expected sequence number
-	return strconv.ParseUint(numbers[0], 10, 64)
+	// Find the position of "expected" and "got" to ensure correct number mapping
+	expectedIdx := strings.Index(str, "expected")
+	gotIdx := strings.Index(str, "got")
+	
+	if expectedIdx == -1 || gotIdx == -1 || expectedIdx > gotIdx {
+		return 0, fmt.Errorf("malformed sequence error message: %s", str)
+	}
+	
+	expected, err := strconv.ParseUint(numbers[0], 10, 64)
+	if err != nil {
+		return 0, fmt.Errorf("failed to parse expected sequence: %w", err)
+	}
+	
+	// Store actual sequence in error message for better debugging
+	actual, err := strconv.ParseUint(numbers[1], 10, 64)
+	if err != nil {
+		return 0, fmt.Errorf("failed to parse actual sequence: %w", err)
+	}
+	
+	return expected, nil
}
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 2ca0feb and 1fd7974.

📒 Files selected for processing (1)
  • app/errors/nonce_mismatch.go (1 hunks)
🔇 Additional comments (2)
app/errors/nonce_mismatch.go (2)

35-37: LGTM! The change correctly addresses the error message format.

The switch from HasPrefix to Contains is appropriate given the actual error message format shown in the PR description.


Line range hint 41-44: Verify the regex implementation

The code relies on a global regexpInt variable that's not visible in the provided context. Let's verify its implementation to ensure it correctly handles the sequence numbers.

✅ Verification successful

The regex implementation is correctly defined and used

The regexpInt variable is properly defined in app/errors/insufficient_gas_price.go with the pattern [0-9]+ to match one or more digits. This implementation is appropriate for extracting sequence numbers from error messages, as it:

  • Correctly matches consecutive digits
  • Is used consistently across both files
  • Handles the expected format of having two numbers in the error message
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for the regexpInt definition
rg -A 2 'regexpInt.*=' 

# Look for any regex-related imports
rg '^import.*regexp'

Length of output: 211


Script:

#!/bin/bash
# Let's get more context around the regex usage in both files
rg -B 2 -A 5 'regexpInt' app/errors/

# Check if there are any other regex patterns defined
rg 'regexp\.MustCompile' app/errors/

# Get the full content of both files to understand the context
cat app/errors/nonce_mismatch.go
cat app/errors/insufficient_gas_price.go

Length of output: 6068

@InoMurko InoMurko changed the title Update nonce_mismatch log parsing fix: nonce_mismatch log parsing Nov 28, 2024
@rootulp
Copy link
Collaborator

rootulp commented Nov 29, 2024

hmm the existing error message parsing seems fragile but why would it not work? If the error is: account sequence mismatch, expected 357545, got 357544: incorrect account sequence then that error message does have the prefix account sequence mismatch.

Before we change the behavior here, I think we need unit test coverage for the expected error messages.

@InoMurko
Copy link
Author

hmm the existing error message parsing seems fragile but why would it not work? If the error is: account sequence mismatch, expected 357545, got 357544: incorrect account sequence then that error message does have the prefix account sequence mismatch.

Before we change the behavior here, I think we need unit test coverage for the expected error messages.

The error starts with: 'broadcast tx error: '

@rootulp
Copy link
Collaborator

rootulp commented Nov 29, 2024

The error starts with: 'broadcast tx error: '

Is that just when the error gets bubbled up to celestia-node? I think that at the point where this error parsing is happening, the error message is just: account sequence mismatch, expected 357545, got 357544: incorrect account sequence. We need unit tests before we can be confident this change has the desired effect.

@InoMurko
Copy link
Author

The error starts with: 'broadcast tx error: '

Is that just when the error gets bubbled up to celestia-node? I think that at the point where this error parsing is happening, the error message is just: account sequence mismatch, expected 357545, got 357544: incorrect account sequence. We need unit tests before we can be confident this change has the desired effect.

This is how the flow goes.
Starts here in celestia-node: https://github.com/celestiaorg/celestia-node/blob/main/state/core_access.go#L255
Then in celestia-app:

resp, err := client.BroadcastPayForBlobWithAccount(ctx, account, blobs, opts...)

return client.broadcastTx(ctx, txBytes, account)

broadcastTxErr := &BroadcastTxError{
TxHash: resp.TxResponse.TxHash,
Code: resp.TxResponse.Code,
ErrorLog: resp.TxResponse.RawLog,
}

broadcast tx error:

return fmt.Sprintf("broadcast tx error: %s", e.ErrorLog)

Given that this piece of code (ParseExpectedSequence, ParseNonceMismatch) is not used and given that a test still exists that did not fail func TestNonceMismatchIntegration(t *testing.T) { I'm unsure what more unit testing would cover, honestly. But if it's absolutely necessary, it can be done.

@InoMurko
Copy link
Author

InoMurko commented Dec 2, 2024

@rootulp let me know if a unit test is needed and if you have any particular flow you'd want to test, given above info!

@rootulp
Copy link
Collaborator

rootulp commented Dec 2, 2024

Given that this piece of code (ParseExpectedSequence, ParseNonceMismatch) is not used

This PR changes the behavior of ParseExpectedSequence so how does it fix the issue if ParseExpectedSequence is not used?

Do you have concrete steps on how I can repro the issue you're observing? It would be helpful for me to repro if you created a bug report issue. Then I can try to review + fix today.

@InoMurko
Copy link
Author

InoMurko commented Dec 2, 2024

Given that this piece of code (ParseExpectedSequence, ParseNonceMismatch) is not used

This PR changes the behavior of ParseExpectedSequence so how does it fix the issue if ParseExpectedSequence is not used?

Do you have concrete steps on how I can repro the issue you're observing? It would be helpful for me to repro if you created a bug report issue. Then I can try to review + fix today.

Haha, good one!
I've shared this in our Telegram group (Caldera <> Celestia).
We run a fork of celestia-node and we run it in our infra (a light node for mocha4 and mainnet).

This piece of code in PR I tried to use at this point in the code
https://github.com/celestiaorg/celestia-node/blob/774be14be2debe5f77ce7f37424e8a074b357b13/state/core_access.go#L258-L268

Our fork has added handling nonces in a similar way that celestia-node handles gas. It checks the network message returned and adjusts the gas.

We handle nonces at the same place. Currently it looks like this:

response, err := ca.client.SubmitPayForBlobWithAccount(ctx, accName, libBlobs, opts...)
		lastErr = err
		// Network min gas price can be updated through governance in app
		// If that's the case, we parse the insufficient min gas price error message and update the gas price
		// broadcast tx error: insufficient minimum gas price for this node; got: 4549 required at least: 4628.000000000000000000: insufficient fee
		//https://github.com/celestiaorg/celestia-node/blob/f5d9b3207dfc6e9a39c0dbb2fa19ef6f96eacf6f/state/core_access.go#L258
		if apperrors.IsInsufficientMinGasPrice(err) {
			minGasPrice, err := ca.queryMinimumGasPrice(ctx)
			if err != nil {
				return nil, fmt.Errorf("insufficient fee: queryMinimumGasPrice() failed: %w", err)
			}
			log.Info("insufficient fee: updated min gas price to ", minGasPrice)
			if minGasPrice > gasPrice {
				gasPrice = minGasPrice
				log.Info("bumped gasPrice from ", gasPrice, " to ", minGasPrice)
			}
			ca.setMinGasPrice(minGasPrice)
			lastErr = err
			continue
		}
//does not work: https://github.com/celestiaorg/celestia-app/pull/4067
		// if err != nil {
		// 	nonce, err := apperrors.ParseExpectedSequence(err.Error())
		// 	if err == nil {
		// 		log.Infof("Adjusting nonce to %d", nonce)
		// 		ca.client.Signer().SetSequence(accName, nonce)
		// 		continue
		// 	} else {
		// 		log.Infof("We were not able to parse ExpectedSequence error:  %w", err)
		// 	}
		// }

		//msg: unexpected wrong sequence error: broadcast tx error: account sequence mismatch, expected 352370, got 352369: incorrect account sequence
		if err != nil && strings.Contains(err.Error(), "incorrect account sequence") {
			regexpInt := regexp.MustCompile(`[0-9]+`)
			numbers := regexpInt.FindAllString(err.Error(), -1)
			log.Infof("Received nonce correction message from the network %w", err.Error())
			if len(numbers) != 2 {
				log.Infof("Could not parse log, will try again...")
				nonce, badNonce, err := parseErrorForNonce(err.Error())
				if err != nil {
					return nil, fmt.Errorf("unexpected wrong sequence error: %w", err)
				}
				log.Infof("Second parsing worked. Adjusting nonce to %d from bad nonce %d", nonce, badNonce)
				ca.client.Signer().SetSequence(accName, nonce)
				continue
			}
			nonce, _ := strconv.ParseUint(numbers[0], 10, 64)
			log.Infof("Adjusting nonce to %d", nonce)
			ca.client.Signer().SetSequence(accName, nonce)
			continue
		}

Hopefully that makes more sense now!

@jcstein
Copy link
Member

jcstein commented Dec 2, 2024

We run a fork of celestia-node and we run it in our infra (a light node for mocha4 and mainnet).

We can only support on official releases from @celestiaorg here, FWIW.

@InoMurko
Copy link
Author

InoMurko commented Dec 2, 2024

We run a fork of celestia-node and we run it in our infra (a light node for mocha4 and mainnet).

We can only support on official releases from @celestiaorg here, FWIW.

Understood. The purpose here is to close the gap.

@jcstein
Copy link
Member

jcstein commented Dec 2, 2024

I think it would make sense to contribute those changes to celestia-node so that others can use it and so it can be upstreamed to prevent you from running a fork. have you made a PR?

@InoMurko
Copy link
Author

InoMurko commented Dec 2, 2024

I think it would make sense to contribute those changes to celestia-node so that others can use it and so it can be upstreamed to prevent you from running a fork. have you made a PR?

Not yet, ran them over @renaynay (Rene Lubov) and was waiting for feedback.

@rootulp
Copy link
Collaborator

rootulp commented Dec 16, 2024

Update: I wonder if celestiaorg/celestia-core#1553 resolves the underlying issue and this PR is no longer necessary 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants