Beat itest [3/3]: fix all itest flakes #9260

yyforyongyu · 2024-11-12T07:13:12Z

Fix all the itest flakes to make sure the blockbeat works as expected. The key results,

All itest flakes are now documented and fixed.
A large decrease in the time taken to run the CI, e.g., for btcd itest, previously it took 45m and now it takes around 18m.

Check #9306 for more context.

In this final PR, we focus on breaking down the large tests into smaller ones, skipping some flaky tests for windows, and minor flake fixes.

coderabbitai · 2024-11-12T07:13:18Z

Important

Review skipped

Auto reviews are limited to specific labels.

🏷️ Labels to auto review (1)

llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

github-actions · 2024-11-12T07:17:16Z

Pull reviewers stats

Stats of the last 30 days for lnd:

User	Total reviews	Time to review	Total comments
guggero 🥇	21 ▀▀▀	1d 2h 36m	26 ▀▀
yyforyongyu 🥈	12 ▀▀	2d 5h 52m ▀	38 ▀▀▀
ellemouton 🥉	6 ▀	16h 51m	19 ▀
bhandras	5 ▀	3h 28m	2
ProofOfKeags	4 ▀	5d 4h 23m ▀▀	14 ▀
dstadulis	4 ▀	2h 13m	7 ▀
ziggie1984	4 ▀	12h 3m	6
ffranr	3	2d 2h 46m ▀	1
Roasbeef	2	3d 12h 37m ▀	6
bitromortac	2	11h 1m	2
saubyk	2	9d 9h 7m ▀▀▀	7 ▀
ViktorTigerstrom	1	3d 23h 47m ▀	4
jharveyb	1	2d 7h 32m ▀	1

guggero · 2024-11-12T11:28:28Z

OMG, what is this sorcery? I don't think I've ever seen this on an lnd PR...

You are my hero, @yyforyongyu!

yyforyongyu · 2024-11-12T14:23:42Z

OMG, what is this sorcery? I don't think I've ever seen this on an lnd PR...

It's a bit of cheating as the unit tests are skipped for me to quickly get the CI results🤓 Plus I know there are two more bugs that can cause itest to fail - one is sql-related and the other is graph, but yeah, at least the flakes are fixed and we should see this as a norm in 2025!

lightninglabs-deploy · 2024-11-26T10:18:26Z

@guggero: review reminder

There's no need to mine 80ish blocks here.

So the test can run faster.

Also removed the duplicate test cases.

To make the CI indicative, we now starting tracking the flaky tests found when running on Windows. As a starting point, rather than ignore the windows CI entirely, we now identify there are cases where lnd can be buggy when running in windows. We should fix the tests in the future, otherwise the windows build should be deleted.

Most of the time we only need to fund the node with given number of UTXOs without concerning the amount, so we add the more efficient funding method as it mines a single block in the end.

guggero

Huge concept ACK!

Will need another round of review to make sure all the refactored test cases still execute the same code paths.

But just the fact that I can now run ALL integration tests locally in just 6.3 minutes is amazing (make itest-parallel tranches=32).

Since things should be way more stable now as well, should we also attempt to bump the number of tranches running in parallel for the CI runs?

guggero · 2024-12-02T12:07:45Z

.github/workflows/main.yml

      - name: run itest
-        run: PATH=$PATH:/tmp/bitcoin/bin make itest-parallel tranches=${{ env.TRANCHES }} backend=bitcoind shuffleseed=${{ github.run_id }}
+        run: PATH=$PATH:/tmp/bitcoin/bin make itest-parallel tranches=${{ env.TRANCHES }} shuffleseed=${{ github.run_id }}


Is btcd faster for these tests or just more stable? Or is the main motivation behind this to remove the additional custom setup above?

guggero · 2024-12-02T12:08:43Z

lntest/harness.go

+
+	// If the number of blocks is less than 40, we consider the test
+	// healthy.
+	if blocksMined < 40 {


guggero · 2024-12-02T12:16:19Z

itest/list_on_test.go

@@ -670,4 +666,5 @@ func init() {
 	allTestCases = append(allTestCases, channelFeePolicyTestCases...)
 	allTestCases = append(allTestCases, walletImportAccountTestCases...)
 	allTestCases = append(allTestCases, basicFundingTestCases...)
+	allTestCases = append(allTestCases, sendToRouteTestCases...)


For all the tests that we add to the list here, should we add a static prefix to their names here and then have more short names in the actual slice where they're defined? Would perhaps help to locate them a bit more easily?
Example (dummy code):

allTestCases = appendPrefixed(allTestCases, "send to route", sendToRouteTestCases...) ... var sendToRouteTestCases = []*lntest.TestCase{ { Name: "single hop sync", }, { Name: "single hop stream", }, { Name: "single hop v2", }, }

But just an idea and non-blocking.

guggero · 2024-12-02T12:19:04Z

itest/list_exclude_test.go

+// be excluded from the test suite atm.
+//
+// TODO(yy): fix these tests and remove them from this list.
+var excludedTestsWindows = []string{


Uff, that's quite a list.... Do we have any idea what causes most of these to fail? Different behavior in network related code?

This was referenced Nov 18, 2024

Beat [4/4]: implement Consumer in chainWatcher #9277

Open

Beat [3/4]: prepare resolvers to handle the blockbeat #9276

Open

Beat [2/4]: implement blockbeat #8894

Merged

yyforyongyu force-pushed the yy-beat-itest-flakes branch from 315ff72 to 2c92455 Compare November 20, 2024 06:16

yyforyongyu force-pushed the yy-beat-itest-optimize branch 2 times, most recently from be8726e to 5a07ffa Compare November 20, 2024 11:32

yyforyongyu force-pushed the yy-beat-itest-flakes branch from 2c92455 to 88350c6 Compare November 20, 2024 12:40

yyforyongyu force-pushed the yy-beat-itest-optimize branch 2 times, most recently from c2aeb68 to 67f9404 Compare November 21, 2024 08:21

yyforyongyu force-pushed the yy-beat-itest-flakes branch from 88350c6 to e259d59 Compare November 21, 2024 13:26

yyforyongyu force-pushed the yy-beat-itest-optimize branch from 67f9404 to ee869a2 Compare November 21, 2024 13:27

yyforyongyu force-pushed the yy-beat-itest-flakes branch from e259d59 to 11587b8 Compare November 21, 2024 16:51

yyforyongyu force-pushed the yy-beat-itest-optimize branch from ee869a2 to 4bdf873 Compare November 21, 2024 16:59

yyforyongyu force-pushed the yy-beat-itest-flakes branch from 11587b8 to a744740 Compare November 21, 2024 16:59

yyforyongyu force-pushed the yy-beat-itest-optimize branch from 4bdf873 to db80ec6 Compare November 21, 2024 17:00

yyforyongyu force-pushed the yy-beat-itest-flakes branch from a744740 to d8e61c8 Compare November 22, 2024 15:49

yyforyongyu force-pushed the yy-beat-itest-optimize branch 2 times, most recently from e24d6b9 to 7fba4ab Compare November 25, 2024 06:29

yyforyongyu force-pushed the yy-beat-itest-flakes branch from d8e61c8 to 43247ea Compare November 25, 2024 07:53

yyforyongyu force-pushed the yy-beat-itest-optimize branch 2 times, most recently from 446296c to 9f93fdd Compare November 25, 2024 09:35

yyforyongyu force-pushed the yy-beat-itest-flakes branch from 43247ea to 269bfd7 Compare November 25, 2024 10:02

yyforyongyu force-pushed the yy-beat-itest-optimize branch 2 times, most recently from 7e1c2d4 to b70b0b0 Compare November 25, 2024 14:06

yyforyongyu mentioned this pull request Nov 25, 2024

htlcswitch+routing: handle nil pointer dereference properly #9303

Open

yyforyongyu force-pushed the yy-beat-itest-flakes branch from 269bfd7 to b1011ac Compare November 26, 2024 08:59

yyforyongyu mentioned this pull request Nov 26, 2024

Reapply #8644 #9242

Open

8 tasks

yyforyongyu added 22 commits November 26, 2024 19:40

itest: optimize blocks mined in testGarbageCollectLinkNodes

bdfa085

There's no need to mine 80ish blocks here.

itest: break remote signer into independent cases

98c7a92

So the test can run faster.

itest: break down channel restore commit types cases

136c6f3

itest: break down utxo selection funding tests

a3473ec

itest: break all multihop test cases

db84d56

itest: break down scid alias channel update tests

006f656

itest: break down open channel fee policy

676b925

itest: break down payment failed tests

6b84e60

itest: break down channel backup restore tests

27b0489

itest: break down wallet import account tests

7a08b78

itest: break down basic funding flow tests

41d1203

itest: break down single hop send to route

c0d284c

Also removed the duplicate test cases.

itest: break down taproot tests

c18e356

itest: break down channel fundmax tests

ce88b8b

itest: further reduce block mined in tests

aace64e

lntest: increase node start timeout and payment benchmark timeout

f2d5b36

lntest: make sure policies are populated in AssertChannelInGraph

41f2387

workflows: use btcd for macOS

b011d8c

itest+lntest: add new method FundNumCoins

be55355

Most of the time we only need to fund the node with given number of UTXOs without concerning the amount, so we add the more efficient funding method as it mines a single block in the end.

lntest: limit the num of blocks mined in each test

87ea4f0

docs: update release notes

7ed3548

yyforyongyu force-pushed the yy-beat-itest-flakes branch from b1011ac to d8d2a54 Compare November 26, 2024 12:02

yyforyongyu force-pushed the yy-beat-itest-optimize branch from 6cee660 to 7ed3548 Compare November 26, 2024 12:02

This was referenced Nov 27, 2024

Reapply 8644 on 9260 #9312

Closed

Reapply 8644 on 9260 #9313

Open

guggero reviewed Dec 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Beat itest [3/3]: fix all itest flakes #9260

Beat itest [3/3]: fix all itest flakes #9260

yyforyongyu commented Nov 12, 2024 •

edited

Loading

coderabbitai bot commented Nov 12, 2024 •

edited

Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

github-actions bot commented Nov 12, 2024

guggero commented Nov 12, 2024

yyforyongyu commented Nov 12, 2024

lightninglabs-deploy commented Nov 26, 2024

guggero left a comment

guggero Dec 2, 2024

guggero Dec 2, 2024

guggero Dec 2, 2024

guggero Dec 2, 2024

Beat itest [3/3]: fix all itest flakes #9260

Are you sure you want to change the base?

Beat itest [3/3]: fix all itest flakes #9260

Conversation

yyforyongyu commented Nov 12, 2024 • edited Loading

coderabbitai bot commented Nov 12, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

github-actions bot commented Nov 12, 2024

Pull reviewers stats

guggero commented Nov 12, 2024

yyforyongyu commented Nov 12, 2024

lightninglabs-deploy commented Nov 26, 2024

guggero left a comment

Choose a reason for hiding this comment

guggero Dec 2, 2024

Choose a reason for hiding this comment

guggero Dec 2, 2024

Choose a reason for hiding this comment

guggero Dec 2, 2024

Choose a reason for hiding this comment

guggero Dec 2, 2024

Choose a reason for hiding this comment

yyforyongyu commented Nov 12, 2024 •

edited

Loading

coderabbitai bot commented Nov 12, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)