Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for re-running failed tests #22

Open
bflad opened this issue Mar 26, 2020 · 1 comment
Open

Support for re-running failed tests #22

bflad opened this issue Mar 26, 2020 · 1 comment

Comments

@bflad
Copy link
Contributor

bflad commented Mar 26, 2020

Certain tests may be flaky and it would be desirable to just re-run the failed tests instead of everything again. For example, this workflow is required now where the re-run command is copy-pasting elements from the results:

$ tctest pr 12530
Discovering tests for pr #12530 (https://github.com/terraform-providers/terraform-provider-aws/pull/12530)...
    TestAccAWSStorageGatewayNfsFileShare
triggering refs/pull/12530/merge for (TestAccAWSStorageGatewayNfsFileShare)...
  bflad@HOSTNAME#Aws_ProviderAwsAlternate
  build 103995 queued: https://HOSTNAME/viewQueued.html?itemId=103995

$ tctest results 103995
--- PASS: TestAccAWSStorageGatewayNfsFileShare_basic (198.28s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_ClientList (265.22s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_DefaultStorageClass (270.50s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_GuessMIMETypeEnabled (287.48s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_KMSKeyArn (303.82s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_tags (315.32s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_Squash (317.48s)
--- FAIL: TestAccAWSStorageGatewayNfsFileShare_KMSEncrypted (716.07s)
--- FAIL: TestAccAWSStorageGatewayNfsFileShare_NFSFileShareDefaults (768.03s)
--- FAIL: TestAccAWSStorageGatewayNfsFileShare_ObjectACL (775.86s)
--- FAIL: TestAccAWSStorageGatewayNfsFileShare_ReadOnly (775.84s)
--- FAIL: TestAccAWSStorageGatewayNfsFileShare_RequesterPays (791.91s)

$ tctest pr 12530 '(TestAccAWSStorageGatewayNfsFileShare_KMSEncrypted|TestAccAWSStorageGatewayNfsFileShare_NFSFileShareDefaults|TestAccAWSStorageGatewayNfsFileShare_ObjectACL|TestAccAWSStorageGatewayNfsFileShare_ReadOnly|TestAccAWSStorageGatewayNfsFileShare_RequesterPays)'
triggering refs/pull/12530/merge for (TestAccAWSStorageGatewayNfsFileShare_KMSEncrypted|TestAccAWSStorageGatewayNfsFileShare_NFSFileShareDefaults|TestAccAWSStorageGatewayNfsFileShare_ObjectACL|TestAccAWSStorageGatewayNfsFileShare_ReadOnly|TestAccAWSStorageGatewayNfsFileShare_RequesterPays)...
  bflad@HOSTNAME#Aws_ProviderAwsAlternate
  build 103998 queued: https://HOSTNAME/viewQueued.html?itemId=103998

It'd be really nice if we could so something like the following instead:

$ tctest run-failed-results 103995
# or
$ tctest pr --failed-only 12530
# or
$ tctest pr --run-failed-results=103995 12530
@bflad
Copy link
Contributor Author

bflad commented Mar 26, 2020

Hacky one liner workaround:

$ tctest pr 12530 $(tctest results 103995 | grep -E -e "^--- FAIL:" | cut -d' ' -f 3 | tr '\n' '|' | sed -e 's/|$//')

I'm guessing there could other potential enhancement ideas here such as:

$ tctest results --failed-only --names-only 103995 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant