Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

command/pr: Support for only running specifically named test patterns (still with auto-detection) #25

Open
bflad opened this issue May 5, 2020 · 0 comments

Comments

@bflad
Copy link
Contributor

bflad commented May 5, 2020

Description

Currently changes to shared files with large amounts of unit testing introduces a large amount of testing noise when running via tctest:

tctest pr 13143
Discovering tests for pr #13143 (https://github.com/terraform-providers/terraform-provider-aws/pull/13143)...
    TestValidateOnceADayWindowFormat
    TestValidateDmsReplicationTaskId
    TestValidateDbParamGroupName
    TestValidateKmsKey
    TestValidateDxConnectionBandWidth
    TestValidateArn
    TestValidateAwsKmsName
    TestValidateCognitoUserPoolEmailVerificationMessage
    TestValidateCognitoUserPoolSmsAuthenticationMessage
    TestValidateIamRoleDescription
    TestValidateTypeStringNullableFloat
    TestValidateLogGroupNamePrefix
    TestValidateDmsReplicationSubnetGroupId
    TestValidateSSMDocumentPermissions
    TestValidateSecurityGroupRuleDescription
    TestValidateLaunchTemplateName
    TestValidateCIDRNetworkAddress
    TestValidateCognitoProviderDeveloperName
    TestResourceAWSElastiCacheReplicationGroupAuthTokenValidation
    TestValidateLambdaPermissionEventSourceToken
    TestValidateEC2AutomateARN
    TestValidateSagemakerName
    TestValidateApiGatewayUsagePlanQuotaSettings
    TestValidateNeptuneEventSubscriptionNamePrefix
    TestValidateDbSubnetGroupName
    TestValidateNeptuneParamGroupNamePrefix
    TestValidateCloudFrontPublicKeyName
    TestValidateAwsKmsGrantName
    TestValidateWafMetricName
    TestValidateCognitoUserGroupName
    TestValidateSecretManagerSecretNamePrefix
    TestValidateLogMetricTransformationName
    TestValidateS3BucketLifecycleTimestamp
    TestValidateSQSFifoQueueName
    TestValidateStepFunctionStateMachineName
    TestValidateAccountAlias
    TestValidateIamRoleProfileNamePrefix
    TestValidateAwsSSMName
    TestValidateCognitoRoleMappingsRulesConfiguration
    TestValidateNeptuneParamGroupName
    TestAccAWSSSMDocument
    TestValidateCloudWatchEventRuleName
    TestValidateAwsAccountId
    TestValidatePolicyStatementId
    TestValidateLogGroupName
    TestValidateIAMPolicyJsonString
    TestValidateEcsPlacementStrategy
    TestValidateDmsCertificateId
    TestValidateNeptuneEventSubscriptionName
    TestValidateDbOptionGroupNamePrefix
    TestValidateCognitoUserPoolEmailVerificationSubject
    TestValidateRoute53ResolverName
    TestValidateLambdaFunctionName
    TestValidateDocDBIdentifier
    TestValidateDbOptionGroupName
    TestValidateOpenIdURL
    TestValidateCognitoSupportedLoginProviders
    TestValidateCognitoIdentityProvidersProviderName
    TestValidateBatchName
    TestValidateCloudFrontPublicKeyNamePrefix
    TestValidateSQSQueueName
    TestValidateDmsEndpointId
    TestValidateCognitoRoleMappingsAmbiguousRoleResolutionAgainstType
    TestValidateCognitoUserPoolId
    TestValidateTypeStringNullableBoolean
    TestValidateLbTargetGroupName
    TestValidateLbTargetGroupNamePrefix
    TestValidateSecretManagerSecretName
    TestAccAWSCognitoIdentityPoolRolesAttachment
    TestValidateCloudWatchDashboardName
    TestValidateLambdaPermissionAction
    TestValidateOnceAWeekWindowFormat
    TestValidateEcsPlacementConstraint
    TestValidateEmrCustomAmiId
    TestValidateIamRoleProfileName
    TestValidateNeptuneSubnetGroupName
    TestValidateDbSubnetGroupNamePrefix
    TestValidateCognitoRoles
    TestValidateElbNamePrefix
    TestValidateNeptuneSubnetGroupNamePrefix
    TestValidateCognitoIdentityPoolName
    TestValidateBatchPrefix
    TestValidateLambdaQualifier
    TestValidateDbEventSubscriptionName
    TestValidateCognitoUserPoolSmsVerificationMessage
    TestValidateLogMetricFilterName
    TestValidateCloudFormationTemplate
    TestValidateDmsReplicationInstanceId
    TestValidateElbName
    TestValidateCognitoIdentityProvidersClientId
    TestValidateAmazonSideAsn
    TestValidateLaunchTemplateId
triggering refs/pull/13143/merge for (TestValidateOnceADayWindowFormat|TestValidateDmsReplicationTaskId|TestValidateDbParamGroupName|TestValidateKmsKey|TestValidateDxConnectionBandWidth|TestValidateArn|TestValidateAwsKmsName|TestValidateCognitoUserPoolEmailVerificationMessage|TestValidateCognitoUserPoolSmsAuthenticationMessage|TestValidateIamRoleDescription|TestValidateTypeStringNullableFloat|TestValidateLogGroupNamePrefix|TestValidateDmsReplicationSubnetGroupId|TestValidateSSMDocumentPermissions|TestValidateSecurityGroupRuleDescription|TestValidateLaunchTemplateName|TestValidateCIDRNetworkAddress|TestValidateCognitoProviderDeveloperName|TestResourceAWSElastiCacheReplicationGroupAuthTokenValidation|TestValidateLambdaPermissionEventSourceToken|TestValidateEC2AutomateARN|TestValidateSagemakerName|TestValidateApiGatewayUsagePlanQuotaSettings|TestValidateNeptuneEventSubscriptionNamePrefix|TestValidateDbSubnetGroupName|TestValidateNeptuneParamGroupNamePrefix|TestValidateCloudFrontPublicKeyName|TestValidateAwsKmsGrantName|TestValidateWafMetricName|TestValidateCognitoUserGroupName|TestValidateSecretManagerSecretNamePrefix|TestValidateLogMetricTransformationName|TestValidateS3BucketLifecycleTimestamp|TestValidateSQSFifoQueueName|TestValidateStepFunctionStateMachineName|TestValidateAccountAlias|TestValidateIamRoleProfileNamePrefix|TestValidateAwsSSMName|TestValidateCognitoRoleMappingsRulesConfiguration|TestValidateNeptuneParamGroupName|TestAccAWSSSMDocument|TestValidateCloudWatchEventRuleName|TestValidateAwsAccountId|TestValidatePolicyStatementId|TestValidateLogGroupName|TestValidateIAMPolicyJsonString|TestValidateEcsPlacementStrategy|TestValidateDmsCertificateId|TestValidateNeptuneEventSubscriptionName|TestValidateDbOptionGroupNamePrefix|TestValidateCognitoUserPoolEmailVerificationSubject|TestValidateRoute53ResolverName|TestValidateLambdaFunctionName|TestValidateDocDBIdentifier|TestValidateDbOptionGroupName|TestValidateOpenIdURL|TestValidateCognitoSupportedLoginProviders|TestValidateCognitoIdentityProvidersProviderName|TestValidateBatchName|TestValidateCloudFrontPublicKeyNamePrefix|TestValidateSQSQueueName|TestValidateDmsEndpointId|TestValidateCognitoRoleMappingsAmbiguousRoleResolutionAgainstType|TestValidateCognitoUserPoolId|TestValidateTypeStringNullableBoolean|TestValidateLbTargetGroupName|TestValidateLbTargetGroupNamePrefix|TestValidateSecretManagerSecretName|TestAccAWSCognitoIdentityPoolRolesAttachment|TestValidateCloudWatchDashboardName|TestValidateLambdaPermissionAction|TestValidateOnceAWeekWindowFormat|TestValidateEcsPlacementConstraint|TestValidateEmrCustomAmiId|TestValidateIamRoleProfileName|TestValidateNeptuneSubnetGroupName|TestValidateDbSubnetGroupNamePrefix|TestValidateCognitoRoles|TestValidateElbNamePrefix|TestValidateNeptuneSubnetGroupNamePrefix|TestValidateCognitoIdentityPoolName|TestValidateBatchPrefix|TestValidateLambdaQualifier|TestValidateDbEventSubscriptionName|TestValidateCognitoUserPoolSmsVerificationMessage|TestValidateLogMetricFilterName|TestValidateCloudFormationTemplate|TestValidateDmsReplicationInstanceId|TestValidateElbName|TestValidateCognitoIdentityProvidersClientId|TestValidateAmazonSideAsn|TestValidateLaunchTemplateId)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 111059 queued: https://--OMITTED--/viewQueued.html?itemId=111059

It would be great if tctest provided a command option to control this behavior, such as --filter or --include-name-pattern or --include-file-pattern, beyond the hardcoded specifying the explicit test name patterns to match the auto-detected pattern. 👍 e.g.

$ tctest pr 13143 --include-name-pattern TestAcc
Discovering tests for pr #13143 (https://github.com/terraform-providers/terraform-provider-aws/pull/13143)...
    TestAccAWSSSMDocument
    TestAccAWSCognitoIdentityPoolRolesAttachment
triggering refs/pull/13143/merge for (TestAccAWSSSMDocument|TestAccAWSCognitoIdentityPoolRolesAttachment)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 111059 queued: https://--OMITTED--/viewQueued.html?itemId=111059
@bflad bflad changed the title Support for only running TestAcc tests Support for only running specifically named tests May 5, 2020
@bflad bflad changed the title Support for only running specifically named tests command/pr: Support for only running specifically named test patterns (still with auto-detection) May 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant