Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Always wait for doCheck to complete before returning #10878

Merged
merged 5 commits into from
Nov 13, 2023

Conversation

ferglor
Copy link
Collaborator

@ferglor ferglor commented Oct 6, 2023

AUTO-6343

We received a notification that some of our tests were flaking in CI: https://chainlink-core.slack.com/archives/C02UPK655SN/p1696419424628789

It seemed to be the case that, in our tests, we were encountering a panic because the test logger was being written to by our application code, after our test had finished.

In our application code, we run doCheck asynchronously, then wait for the result channel to receive a value, or for the parent context to be cancelled.

I think this particular flake is caused by the cancellation of the context while doCheck is still executing. When the context is cancelled, we return, which subsequently allows the test to execute until completion. However, it's possible that doCheck is still running in the background, and writing to the test logger.

This change uses a wait group to wait for doCheck to complete, even when the parent context is cancelled. It feels like an inappropriate solution given that we seem to anticipate context cancellations in our code, but the problem is that doCheck and its child functions do not check for context cancellations, so the simpler solution is to instead wait for doCheck to finish execution before returning.

@ferglor ferglor requested a review from a team as a code owner October 6, 2023 16:10
@github-actions
Copy link
Contributor

github-actions bot commented Oct 6, 2023

I see that you haven't updated any CHANGELOG files. Would it make sense to do so?

@ferglor ferglor requested a review from amirylm October 10, 2023 13:42
@ferglor ferglor force-pushed the spike/registry-lookup-testflake branch 2 times, most recently from b228dbf to 312aa37 Compare October 17, 2023 23:52
@ferglor ferglor changed the title Spike: Always wait for doCheck to complete before returning Always wait for doCheck to complete before returning Oct 18, 2023
@ferglor ferglor force-pushed the spike/registry-lookup-testflake branch from 0f56b60 to 01c9801 Compare October 18, 2023 07:53
@ferglor ferglor requested a review from amirylm October 18, 2023 21:36
amirylm
amirylm previously approved these changes Oct 20, 2023
Use threadctrl to avail of context cancellation but still rely on a wait group to blocl until the lookups finish
Update tests
@ferglor ferglor force-pushed the spike/registry-lookup-testflake branch from 77e98c1 to c6e09f1 Compare November 9, 2023 15:28
@cl-sonarqube-production
Copy link

@ferglor ferglor added this pull request to the merge queue Nov 13, 2023
Merged via the queue into develop with commit de50273 Nov 13, 2023
86 checks passed
@ferglor ferglor deleted the spike/registry-lookup-testflake branch November 13, 2023 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants