Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Queries to services do not return all resources because paging is not being used #8

Closed
vinelias opened this issue Mar 8, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@vinelias
Copy link

vinelias commented Mar 8, 2023

Describe the bug
When working on accounts with large numbers of resources in a given service, only a portion of the resources are returned in the query. For example, if we have more than 100 IAM roles, the query will return only the first 100, as this is the default limit of the list_roles method page of the iam service in boto3. (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam/client/list_roles.html). This goes for other resources like IAM Policies, secrets in Secrets Manager, KMS Keys, SNS Topics, etc...

To Reproduce
Take an account over 100 IAM roles and start a Resource-Based Policies scan. You will only see the first 100 IAM roles.

Expected behavior
View all IAM roles in the account

Please complete the following information about the solution:

  • [v1.0.2] Version:

To get the version of the solution, you can look at the description of the created CloudFormation stack.

For example, "(SO0217) - The AWS CloudFormation hub template for deployment of the Account Assessment for AWS Organisations, Version: v1.0.0".

  • [us-east-1] Region: [e.g. us-east-1]
  • [no] Was the solution modified from the version published on this repository?
  • [n/a] If the answer to the previous question was yes, are the changes available on GitHub?
  • [n/a] Have you checked your service quotas for the sevices this solution uses?
  • [no] Were there any errors in the CloudWatch Logs?

Screenshots
If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context
Add any other context about the problem here.
I collected the information I needed using a personal script and applied paging whenever the resource allowed. e.g:

#Get IAM roles
...
    resource_list = []
    paginator = client.get_paginator('list_roles')
    page_iterator = paginator.paginate()
    for page in page_iterator:
      for item in page.get('Roles'):
        if item is not None:
          resource_list.append(item)
....
if resource_list is not None:
    try:
      for resource in resource_list:
....
@vinelias vinelias added the bug Something isn't working label Mar 8, 2023
@vinelias
Copy link
Author

vinelias commented Mar 8, 2023

My bad! This issue was opened erroneously. The solution is working as expected, the test was done wrong.

@vinelias vinelias closed this as not planned Won't fix, can't repro, duplicate, stale Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant