Skip to content

Commit

Permalink
Check empty array before terminating/deleting (aws-observability#147)
Browse files Browse the repository at this point in the history
*Issue description:*
Check empty array before deleting ASGs or terminating instances in
resource clean up to avoid errors:
```
Error terminating instances: An error occurred (InvalidParameterCombination) when calling the TerminateInstances operation: No instances specified
```

*Description of changes:*

*Ensure you've run the following tests on your changes and include the
link below:*
To do so, create a `test.yml` file with `name: Test` and workflow
description to test your changes, then remove the file for your PR. Link
your test run in your PR description. This process is a short term
solution while we work on creating a staging environment for testing.

NOTE: TESTS RUNNING ON A SINGLE EKS CLUSTER CANNOT BE RUN IN PARALLEL.
See the
[needs](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idneeds)
keyword to run tests in succession.
- Run Java EKS on `e2e-playground` in us-east-1 and eu-central-2
- Run Python EKS on `e2e-playground` in us-east-1 and eu-central-2
- Run metric limiter on EKS cluster `e2e-playground` in us-east-1 and
eu-central-2
- Run EC2 tests in all regions
- Run K8s on a separate K8s cluster (check IAD test account for master
node endpoints; these will change as we create and destroy clusters for
OS patching)

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
  • Loading branch information
bjrara authored Aug 1, 2024
1 parent 61bc44a commit b24c80d
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 6 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/resource-cleanup.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
role-to-assume: arn:aws:iam::${{ env.ACCOUNT_ID }}:role/${{ secrets.RESOURCE_CLEANER_ROLE_NAME }}
aws-region: ${{ matrix.aws-region }}

- name: Cleanup EC2 instances
- name: Cleanup EC2 instances and ASGs
working-directory: .github/workflows/util/clean/ec2_instance_cleanup
env:
AWS_DEFAULT_REGION: ${{ matrix.aws-region }}
Expand Down
15 changes: 10 additions & 5 deletions .github/workflows/util/clean/ec2_instance_cleanup/cleaner.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ def _delete_autoscaling_groups(auto_scaling_groups):
logging.info("===== Response for delete autoscaling group request =====")
logging.info(response)
except Exception as e:
logging.info(f"Error terminating instances: {e}")
logging.info(f"Error deleting groups: {e}")

def _is_active(asg):
for instance in asg['Instances']:
Expand Down Expand Up @@ -191,13 +191,18 @@ def _terminate_instances(instances_to_terminate):
instances = _get_instances_to_terminate()

if len(groups) == 0 and len(instances) == 0:
logging.info("No resource to terminate")
logging.info("No resource to clean up")
exit(0)

report_successful = _prepare_report_and_upload(groups, instances)
if not report_successful:
logging.error("Failed to prepare report and upload. Aborting termination of instances.")
logging.error("Failed to prepare report and upload. Aborting resource clean up.")
exit(1)

_delete_autoscaling_groups(groups)
_terminate_instances(instances)
if len(groups) > 0:
logging.info("Deleting autoscaling groups...")
_delete_autoscaling_groups(groups)

if len(instances) > 0:
logging.info("Terminating instances...")
_terminate_instances(instances)

0 comments on commit b24c80d

Please sign in to comment.