Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed flaky TestGameServerWithPortsMappedToMultipleContainers #1458

Merged
merged 1 commit into from
Apr 11, 2020

Conversation

akremsa
Copy link
Contributor

@akremsa akremsa commented Apr 9, 2020

Closes #1450
I've run this test more than 100 times and found that both pings can fail.

Fix proof:
Game port №1:
image

Game port №2:
image

Suggestion: Is there any opportunity to stop tests immediately by pressing ctr+c/ctr+z if we run them inside a container? It can take some time for some goroutines to finish their work.

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 0996fc2a-1b0a-4e87-8ed4-603949296d42

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1458/head:pr_1458 && git checkout pr_1458
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-0c9c21a

test/e2e/gameserver_test.go Outdated Show resolved Hide resolved
test/e2e/gameserver_test.go Outdated Show resolved Hide resolved
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 6c6dd98a-6a7c-4d36-96f3-6306c2196928

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1458/head:pr_1458 && git checkout pr_1458
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-8cb9371

@roberthbailey roberthbailey requested review from markmandel and removed request for roberthbailey and dzlier-gcp April 9, 2020 17:31

go func() {
expectedMsg := "Ping 1"
errPoll := wait.PollImmediate(interval, timeOut, func() (done bool, err error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love the polling - seems like a good solution!

pollCh1 := make(chan error)
pollCh2 := make(chan error)

go func() {
Copy link
Collaborator

@markmandel markmandel Apr 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason this couldn't be 2 polling statements, one after another? (i.e. don't run it in goroutines)

Then it would make this test much simpler, and remove the need for channels. I don't think we have to test for both UDP endpoints at the same time.

WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main idea why I've introduced 2 goroutines is that in a worst-case scenario if we ping ports one by one, this single test will run 2 mins at least - there are 2 timeouts, 60 sec for each ping. According to my tests, there is a little chance of this scenario, but I decided to take it into account. I've tried to make this code as simple as possible.
Do you think we need to call polling statements one after another anyway?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since failure should be an exception not the regular path, I would have them one after the other, for sake of simplicity of the testing code. This would make it easier to maintain going forward.

One optimisation you could make that would cut things down time in failure scenarios, is that if the first one fails, do a assert.FailNow(); to fast fail the test and stop the second one from even starting.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, good suggestion!

@akremsa
Copy link
Contributor Author

akremsa commented Apr 10, 2020

@markmandel Applied your review notes.

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: a4627933-f8f5-4ccd-a7c3-cd01a91ac0b9

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1458/head:pr_1458 && git checkout pr_1458
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-c7e6144

return err == nil && strings.Contains(res, expectedMsg2), nil
})

assert.Nil(t, errPoll, "expected no errors after polling a port: %s", secondPort)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
assert.Nil(t, errPoll, "expected no errors after polling a port: %s", secondPort)
assert.NoError(t, errPoll, "expected no errors after polling a port: %s", secondPort)

Just displays the details of the passed in error message. Other than that, this looks good to go 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: 8845fad4-d53d-4c02-a0db-45bb77a62017

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: b8054085-b471-4c77-b955-6d763f2f3b42

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: c054af4a-aa0e-44fe-9f91-a209731d821e

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1458/head:pr_1458 && git checkout pr_1458
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-4b04b15

@markmandel markmandel added area/tests Unit tests, e2e tests, anything to make sure things don't break kind/bug These are bugs. labels Apr 11, 2020
@google-oss-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: akremsa, markmandel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@markmandel markmandel merged commit c9c5399 into googleforgames:master Apr 11, 2020
@markmandel markmandel added this to the 1.5.0 milestone Apr 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved area/tests Unit tests, e2e tests, anything to make sure things don't break kind/bug These are bugs. lgtm size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flaky: TestGameServerWithPortsMappedToMultipleContainers
6 participants