Rebase release/v1.32.x (#656)

* E2E Test: Ensure the use of IMDSv2 in EC2 instances (#621) * Add e2e canary to public preview regions (#623) * Fix trace validation error follow up fix (#626) * Fix Terrform Destroy Error on EKS Canary (#628) * fix-e2e-eks-terraform-destroy-error * Add region as parameter to terraform destroy * Bump nebula.release from 17.2.2 to 18.0.6 (#631) Bumps nebula.release from 17.2.2 to 18.0.6. --- updated-dependencies: - dependency-name: nebula.release dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/setup-java from 3 to 4 (#629) Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](actions/setup-java@v3...v4) --- updated-dependencies: - dependency-name: actions/setup-java dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump hashicorp/setup-terraform from 2 to 3 (#586) Bumps [hashicorp/setup-terraform](https://github.com/hashicorp/setup-terraform) from 2 to 3. - [Release notes](https://github.com/hashicorp/setup-terraform/releases) - [Changelog](https://github.com/hashicorp/setup-terraform/blob/main/CHANGELOG.md) - [Commits](hashicorp/setup-terraform@v2...v3) --- updated-dependencies: - dependency-name: hashicorp/setup-terraform dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rust from 1.73 to 1.74 (#611) Bumps rust from 1.73 to 1.74. --- updated-dependencies: - dependency-name: rust dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/setup-node from 3 to 4 (#574) Bumps [actions/setup-node](https://github.com/actions/setup-node) from 3 to 4. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v3...v4) --- updated-dependencies: - dependency-name: actions/setup-node dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump tempfile from 3.8.0 to 3.8.1 in /tools/cp-utility (#585) Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.8.0 to 3.8.1. - [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md) - [Commits](https://github.com/Stebalien/tempfile/commits) --- updated-dependencies: - dependency-name: tempfile dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Provide aws-region for the e2e test in worklow (#643) * Provide aws-region for the e2e test in worklow * Update region to us-east-1 and add concurrency * Revert "Provide aws-region for the e2e test in worklow (#643)" (#645) This reverts commit 44b5b68. * E2E Testing: Add concurrency tag to test in main build and nightly build (#646) * Use aws-region in the workflow (#649) * Add Retry Mechanism to E2E EKS Terraform Deployment (#634) * Add Retry Mechanism to E2E EKS Terraform Deployment * Add Extra Comments * Call Test APIs First before Validation * Add clean-app-signals to retry logic * Change App Signal Download Directory and modify if statement for validation * Modify while loop and refactor code * Dynamic input RPM link by region setting (#647) * Dynamic input RPM link by region setting * Remove unneeded env variable * Fix an issue in echo shell command * Revert previous wrong 'fix' regarding variable call * Add Retry Mechanism to E2E EC2 Terraform Deployment (#635) * Add Retry Mechanism to E2E EC2 Terraform Deployment * Add Extra Comments * Refactor code * Change App Signals Directory (#650) * change dep config to compileOnly to fix high cardinality metrics (#651) * E2E Testing: Fix EKS test candidate image override (#652) This change checks if there is an adot image passed to the workflow and patches the App Signals deployment to update the image and restarts the cloudwatch pods. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Mahad Janjua <[email protected]> Co-authored-by: Harry <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Vasi Vasireddy <[email protected]> Co-authored-by: XinRan Zhang <[email protected]> Co-authored-by: Mengyi Zhou (bjrara) <[email protected]>
aws-observability · Dec 15, 2023 · 2283eda · 2283eda
1 parent 24bfb35
commit 2283eda
Show file tree

Hide file tree

Showing 39 changed files with 442 additions and 245 deletions.
diff --git a/.github/workflows/appsignals-e2e-ec2-canary-test.yml b/.github/workflows/appsignals-e2e-ec2-canary-test.yml
@@ -14,7 +14,12 @@ permissions:
 
 jobs:
   e2e-canary-test:
+    strategy:
+      fail-fast: false
+      matrix:
+        aws-region: ['us-east-1', 'us-east-2', 'eu-west-1', 'ap-northeast-1', 'ap-southeast-2']
     uses: ./.github/workflows/appsignals-e2e-ec2-test.yml
     secrets: inherit
     with:
+      aws-region: ${{ matrix.aws-region }}
       caller-workflow-name: 'appsignals-e2e-ec2-canary-test'
diff --git a/.github/workflows/appsignals-e2e-ec2-test.yml b/.github/workflows/appsignals-e2e-ec2-test.yml
@@ -5,6 +5,9 @@ name: App Signals Enablement E2E Testing - EC2 Use Case
 on:
   workflow_call:
     inputs:
+      aws-region:
+        required: true
+        type: string
       caller-workflow-name:
         required: true
         type: string
@@ -14,11 +17,10 @@ permissions:
   contents: read
 
 env:
-  AWS_DEFAULT_REGION: us-east-1
+  AWS_DEFAULT_REGION: ${{ inputs.aws-region }} # Used by terraform and AWS CLI commands
   TEST_ACCOUNT: ${{ secrets.APP_SIGNALS_E2E_TEST_ACC }}
   SAMPLE_APP_FRONTEND_SERVICE_JAR: "s3://aws-appsignals-sample-app/main-service.jar"
   SAMPLE_APP_REMOTE_SERVICE_JAR: "s3://aws-appsignals-sample-app/remote-service.jar"
-  APP_SIGNALS_CW_AGENT_RPM: "https://amazoncloudwatch-agent-us-east-1.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm"
   APP_SIGNALS_ADOT_JAR: "https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest/download/aws-opentelemetry-agent.jar"
   METRIC_NAMESPACE: AppSignals
   LOG_GROUP_NAME: /aws/appsignals/generic
@@ -31,6 +33,15 @@ jobs:
         with:
           fetch-depth: 0
 
+      - name: Set CW Agent RPM environment variable
+        run: |
+          if [ ${{ env.AWS_DEFAULT_REGION }} == "us-east-1" ]; then
+            echo APP_SIGNALS_CW_AGENT_RPM="https://amazoncloudwatch-agent-us-east-1.s3.amazonaws.com/amazon_linux/amd64/1.300031.0b313/amazon-cloudwatch-agent.rpm" >> $GITHUB_ENV
+          else
+            echo APP_SIGNALS_CW_AGENT_RPM="https://amazoncloudwatch-agent-${{ env.AWS_DEFAULT_REGION }}.s3.${{ env.AWS_DEFAULT_REGION }}.amazonaws.com/amazon_linux/amd64/1.300031.0b313/amazon-cloudwatch-agent.rpm" >> $GITHUB_ENV
+          fi
+          
+
       - name: Generate testing id
         run: echo TESTING_ID="${{ github.run_id }}-${{ github.run_number }}" >> $GITHUB_ENV
 
@@ -41,45 +52,88 @@ jobs:
           aws-region: ${{ env.AWS_DEFAULT_REGION }}
 
       - name: Set up terraform
-        uses: hashicorp/setup-terraform@v2
+        uses: hashicorp/setup-terraform@v3
         with:
           terraform_wrapper: false
 
-      - name: Deploy sample app via terraform
+      - name: Deploy sample app via terraform and wait for endpoint to come online
         working-directory: testing/terraform/ec2
         run: |
           terraform init
           terraform validate
-          terraform apply -auto-approve \
-            -var="aws_region=${{ env.AWS_DEFAULT_REGION }}" \
-            -var="test_id=${{ env.TESTING_ID }}" \
-            -var="sample_app_jar=${{ env.SAMPLE_APP_FRONTEND_SERVICE_JAR }}" \
-            -var="sample_remote_app_jar=${{ env.SAMPLE_APP_REMOTE_SERVICE_JAR }}" \
-            -var="cw_agent_rpm=${{ env.APP_SIGNALS_CW_AGENT_RPM }}" \
-            -var="adot_jar=${{ env.APP_SIGNALS_ADOT_JAR }}"
+          
+          # Attempt to deploy the sample app on an EC2 instance and wait for its endpoint to come online. 
+          # There may be occasional failures due to transitivity issues, so try up to 2 times. 
+          # deployment_failed of 0 indicates that both the terraform deployment and the endpoint are running, while 1 indicates
+          # that it failed at some point
+          retry_counter=0
+          max_retry=2
+          while [ $retry_counter -lt $max_retry ]; do
+            echo "Attempt $retry_counter"
+            deployment_failed=0
+            terraform apply -auto-approve \
+              -var="aws_region=${{ env.AWS_DEFAULT_REGION }}" \
+              -var="test_id=${{ env.TESTING_ID }}" \
+              -var="sample_app_jar=${{ env.SAMPLE_APP_FRONTEND_SERVICE_JAR }}" \
+              -var="sample_remote_app_jar=${{ env.SAMPLE_APP_REMOTE_SERVICE_JAR }}" \
+              -var="cw_agent_rpm=${{ env.APP_SIGNALS_CW_AGENT_RPM }}" \
+              -var="adot_jar=${{ env.APP_SIGNALS_ADOT_JAR }}" \
+            || deployment_failed=$?
+          
+            if [ $deployment_failed -eq 1 ]; then
+              echo "Terraform deployment was unsuccessful. Will attempt to retry deployment."
+            fi
+          
+            # If the deployment_failed is still 0, then the terraform deployment succeeded and now try to connect to the endpoint.
+            # Attempts to connect will be made for up to 10 minutes
+            if [ $deployment_failed -eq 0 ]; then
+              echo "Attempting to connect to the endpoint"
+              sample_app_endpoint=http://$(terraform output sample_app_main_service_public_dns):8080
+              attempt_counter=0
+              max_attempts=60
+              until $(curl --output /dev/null --silent --head --fail $(echo "$sample_app_endpoint" | tr -d '"')); do
+                if [ ${attempt_counter} -eq ${max_attempts} ];then
+                  echo "Failed to connect to endpoint. Will attempt to redeploy sample app."
+                  deployment_failed=1
+                  break
+                fi
+          
+                printf '.'
+                attempt_counter=$(($attempt_counter+1))
+                sleep 10
+              done
+            fi
+          
+            # If the success is 1 then either the terraform deployment or the endpoint connection failed, so first destroy the
+            # resources created from terraform and try again.
+            if [ $deployment_failed -eq 1 ]; then
+              echo "Destroying terraform"
+              terraform destroy -auto-approve \
+                -var="test_id=${{ env.TESTING_ID }}" 
+          
+              retry_counter=$(($retry_counter+1))
+            else
+              # If deployment succeeded, then exit the loop
+              break
+            fi
+          
+            if [ $retry_counter -eq $max_retry ]; then
+              echo "Max retry reached, failed to deploy terraform and connect to the endpoint. Exiting code"
+              exit 1
+            fi
+          done
+
+      - name: Get the ec2 instance ami id
+        run: |
+          echo "EC2_INSTANCE_AMI=$(terraform output ec2_instance_ami)" >> $GITHUB_ENV
+        working-directory: testing/terraform/ec2
 
       - name: Get the sample app endpoint
         run: |
           echo "MAIN_SERVICE_ENDPOINT=$(terraform output sample_app_main_service_public_dns):8080" >> $GITHUB_ENV
           echo "REMOTE_SERVICE_IP=$(terraform output sample_app_remote_service_public_ip)" >> $GITHUB_ENV
         working-directory: testing/terraform/ec2
 
-      - name: Wait for app endpoint to come online
-        id: endpoint-check
-        run: |
-          attempt_counter=0
-          max_attempts=30
-          until $(curl --output /dev/null --silent --head --fail http://${{ env.MAIN_SERVICE_ENDPOINT }}); do
-            if [ ${attempt_counter} -eq ${max_attempts} ];then
-              echo "Max attempts reached"
-              exit 1
-            fi
-
-            printf '.'
-            attempt_counter=$(($attempt_counter+1))
-            sleep 10
-          done
-
       # This steps increases the speed of the validation by creating the telemetry data in advance
       - name: Call all test APIs
         continue-on-error: true
@@ -103,6 +157,7 @@ jobs:
           --service-name sample-application-${{ env.TESTING_ID }}
           --remote-service-name sample-remote-application-${{ env.TESTING_ID }}
           --request-body ip=${{ env.REMOTE_SERVICE_IP }}
+          --instance-ami ${{ env.EC2_INSTANCE_AMI }}
           --rollup'
 
       - name: Validate generated metrics
@@ -119,6 +174,7 @@ jobs:
           --service-name sample-application-${{ env.TESTING_ID }}
           --remote-service-name sample-remote-application-${{ env.TESTING_ID }}
           --request-body ip=${{ env.REMOTE_SERVICE_IP }}
+          --instance-ami ${{ env.EC2_INSTANCE_AMI }}
           --rollup'
 
       - name: Validate generated traces
@@ -135,6 +191,7 @@ jobs:
           --service-name sample-application-${{ env.TESTING_ID }}
           --remote-service-name sample-remote-application-${{ env.TESTING_ID }}
           --request-body ip=${{ env.REMOTE_SERVICE_IP }}
+          --instance-ami ${{ env.EC2_INSTANCE_AMI }}
           --rollup'
 
       - name: Publish metric on test result
@@ -163,4 +220,4 @@ jobs:
         working-directory: testing/terraform/ec2
         run: |
           terraform destroy -auto-approve \
-            -var="test_id=${{ env.TESTING_ID }}" 
+            -var="test_id=${{ env.TESTING_ID }}"
diff --git a/.github/workflows/appsignals-e2e-eks-canary-test.yml b/.github/workflows/appsignals-e2e-eks-canary-test.yml
@@ -17,9 +17,14 @@ permissions:
   contents: read
 
 jobs:
-  e2e-canary-test:
+  e2e-test:
+    strategy:
+      fail-fast: false
+      matrix:
+        aws-region: ['us-east-1', 'us-east-2', 'eu-west-1', 'ap-northeast-1', 'ap-southeast-2']
     uses: ./.github/workflows/appsignals-e2e-eks-test.yml
     secrets: inherit
     with:
+      aws-region: ${{ matrix.aws-region }}
       test-cluster-name: 'e2e-canary-test'
       caller-workflow-name: 'appsignals-e2e-eks-canary-test'