Merge pull request #21 from kmcquade/fix/botocore-credentials-loading

Fix credentials loading for download command. Set required minimum Policy Sentry version
salesforce · May 5, 2020 · 802ad65 · 802ad65
2 parents 47c20de + 10d7a71
commit 802ad65
Show file tree

Hide file tree

Showing 9 changed files with 82 additions and 169 deletions.
diff --git a/.gitignore b/.gitignore
@@ -9,6 +9,7 @@ default-iam-report.csv
 private/default-iam-results.json
 default-results-summary.csv
 private/*
+current.json
 
 Pipfile.lock
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,10 @@
 # CHANGELOG
 
+## 0.0.10 (2020-05-05)
+* Removed the recursive credentials method from the `download` command.
+* Fixed occasional installation error occurring from outdated Policy Sentry versions.
+* Fixed instructions for the `download` command.
+
 ## 0.0.9 (2020-05-03)
 * HTML report now always shows Trust Policies for Roles, even if they do not allow assumption from a Compute Service. This can help assessors with triaging and pentesters for targeting.
 

diff --git a/README.md b/README.md
@@ -101,16 +101,11 @@ pip3 install --user cloudsplaining
 
 We can scan an entire AWS account and generate reports. To do this, we leverage the AWS IAM [get-account-authorization-details](https://docs.aws.amazon.com/cli/latest/reference/iam/get-account-authorization-details.html) API call, which downloads a large JSON file (around 100KB per account) that contains all of the IAM details for the account. This includes data on users, groups, roles, customer-managed policies, and AWS-managed policies.
 
-* To do this, set your AWS access keys as environment variables:
+* You must have AWS credentials configured that can be used by the CLI.
 
-```bash
-export AWS_ACCESS_KEY_ID=...
-export AWS_SECRET_ACCESS_KEY=...
-# If you are using MFA or STS; optional but highly recommended
-export AWS_SESSION_TOKEN=...
-```
+* You must have the privileges to run [iam:GetAccountAuthorizationDetails](https://docs.aws.amazon.com/IAM/latest/APIReference/API_GetAccountAuthorizationDetails.html). The `arn:aws:iam::aws:policy/SecurityAudit` policy includes this, as do many others that allow Read access to the IAM Service.
 
-* Then run `cloudsplaining`'s `download` command:
+* To download the account authorization details, ensure you are authenticated to AWS, then run `cloudsplaining`'s `download` command:
 
 ```bash
 cloudsplaining download
@@ -119,10 +114,10 @@ cloudsplaining download
 * If you prefer to use your `~/.aws/credentials` file instead of environment variables, you can specify the profile name:
 
 ```bash
-cloudsplaining download --profile default
+cloudsplaining download --profile myprofile
 ```
 
-It will download a file titled `default.json` in your current directory.
+It will download a JSON file in your current directory that contains your account authorization detail information.
 
 #### Create Exclusions file
 
@@ -279,8 +274,6 @@ Actions: ecr:BatchDeleteImage, ecr:CompleteLayerUpload, ecr:CreateRepository, ec
 cloudsplaining download
 # Download from a specific profile
 cloudsplaining download --profile someprofile
-# Download authorization details for **all** of your AWS profiles
-cloudsplaining download --profile all
 
 # Scan Authorization details
 cloudsplaining scan --input default.json
@@ -302,14 +295,18 @@ No, it will only scan policies that are attached to IAM principals.
 
 Not by default. If you want to do this, specify the `--include-non-default-policy-versions` flag. Note that the `scan` tool does not currently operate on non-default versions.
 
-**I followed the installation instructions but can't execute the program via command line. What do I do?**
+**I followed the installation instructions but can't execute the program via command line at all. What do I do?**
 
 This is likely an issue with your PATH. Your PATH environment variable is not considering the binary packages installed by `pip3`. On a Mac, you can likely fix this by entering the command below, depending on the versions you have installed. YMMV.
 
 ```bash
 export PATH=$HOME/Library/Python/3.7/bin/:$PATH
 ```
 
+**I followed the installation instructions but I am receiving a `ModuleNotFoundError` that says `No module named policy_sentry.analysis.expand`. What should I do?**
+
+Try upgrading to the latest version of Cloudsplaining. This error was fixed in version 0.0.10.
+
 ## References
 
 * [Policy Sentry](https://github.com/salesforce/policy_sentry/) by [Kinnaird McQuade](https://twitter.com/kmcquade3) at Salesforce

diff --git a/cloudsplaining/bin/cloudsplaining b/cloudsplaining/bin/cloudsplaining
@@ -7,7 +7,7 @@
 """
     Cloudsplaining is an AWS IAM Assessment tool that identifies violations of least privilege and generates a risk-prioritized HTML report with a triage worksheet.
 """
-__version__ = "0.0.9"
+__version__ = "0.0.10"
 import click
 from cloudsplaining import command
 

diff --git a/cloudsplaining/command/download.py b/cloudsplaining/command/download.py
@@ -8,7 +8,6 @@
 import os
 import json
 import logging
-import configparser
 from pathlib import Path
 import boto3
 import click
@@ -26,8 +25,7 @@
 @click.option(
     "--profile",
     type=str,
-    default="default",
-    required=True,
+    required=False,
     help="Specify 'all' to authenticate to AWS and analyze *all* existing IAM policies. Specify a non-default "
     "profile here. Defaults to the 'default' profile.",
 )
@@ -37,12 +35,6 @@
     default=Path.cwd(),
     help="Path to store the output. Defaults to current directory.",
 )
-@click.option(
-    "--credentials-file",
-    type=click.Path(exists=False),
-    help="Path to the AWS credentials file.",
-    default=str(Path.home()) + "/.aws/credentials",
-)
 @click.option(
     "--include-non-default-policy-versions",
     is_flag=True,
@@ -51,82 +43,23 @@
     " Note that this will dramatically increase the size of the downloaded file.",
 )
 @click_log.simple_verbosity_option(logger)
-def download(profile, output, credentials_file, include_non_default_policy_versions):
+def download(profile, output, include_non_default_policy_versions):
     """
     Runs aws iam get-authorization-details on all accounts specified in the aws credentials file, and stores them in
     account-alias.json
     """
-    # Just default profile
-    if profile == "default":
-        profiles = ["default"]
-        print("profile: default")
-    # Get all profiles
-    elif profile == "all":
-        profiles = get_list_of_aws_profiles(credentials_file)
-        print("profile: all profiles")
-        print(profiles)
-    else:
-        credentials_file_profiles = get_list_of_aws_profiles(credentials_file)
-        # If it exists in ~/.aws/credentials,
-        if profile in credentials_file_profiles:
-            print(f"profile: {profile}")
-            profiles = [profile]
-        else:
-            raise Exception(
-                "The profile %s is not in the ~/.aws/credentials file", profile
-            )
-    for profile in profiles:
-        print("Running get_account_authorization_details for profile: ", profile)
-        get_account_authorization_details(
-            profile, output, include_non_default_policy_versions
-        )
-
-
-def get_account_authorization_details(
-    profile, output, include_non_default_policy_versions=False
-):
-    """
-    Run aws iam get-account-authorization-details and store locally.
-
-    :param profile: Name of the profile in the AWS Credentials file
-    :param output: The path of a directory to store the results.
-    :param include_non_default_policy_versions: When downloading AWS managed policy documents, also include the non-default policy versions. Note that this will dramatically increase the size of the downloaded file.
-    :return:
-    """
-
-    def get_session_via_environment_variables():
-        aws_access_key_id = os.getenv("AWS_ACCESS_KEY_ID")
-        aws_secret_access_key = os.getenv("AWS_SECRET_ACCESS_KEY")
-        aws_session_token = os.getenv("AWS_SESSION_TOKEN")
-        if aws_access_key_id and aws_secret_access_key and aws_session_token:
-            session = boto3.Session(
-                aws_access_key_id=aws_access_key_id,
-                aws_secret_access_key=aws_secret_access_key,
-                aws_session_token=aws_session_token,
-            )
-            return session
-        elif aws_access_key_id and aws_secret_access_key and not aws_session_token:
-            print(
-                "It looks like you are authenticating with static credentials rather than temporary credentials. "
-                "Static credentials usage is a security concern. We suggest you use MFA to generate short lived "
-                "credentials and set those as environment variables instead."
-            )
-            session = boto3.Session(
-                aws_access_key_id=aws_access_key_id,
-                aws_secret_access_key=aws_secret_access_key,
-            )
-            return session
-        else:
-            return False
+    default_region = "us-east-1"
+    session_data = {"region_name": default_region}
 
-    config = Config(connect_timeout=5, retries={"max_attempts": 10})
-    if get_session_via_environment_variables():
-        print("AWS Credentials found in Environment variables.")
-        boto3_session = get_session_via_environment_variables()
+    if profile:
+        session_data["profile_name"] = profile
+        output_filename = os.path.join(output, f"{profile}.json")
     else:
-        boto3_session = boto3.Session(profile_name=profile)
+        output_filename = "default.json"
 
-    iam_client = boto3_session.client("iam", config=config)
+    session = boto3.Session(**session_data)
+    config = Config(connect_timeout=5, retries={"max_attempts": 10})
+    iam_client = session.client("iam", config=config)
 
     results = {
         "UserDetailList": [],
@@ -182,46 +115,9 @@ def get_session_via_environment_variables():
                     }
                     results["Policies"].append(entry)
 
-    filename = os.path.join(output, f"{profile}.json")
-    if os.path.exists(filename):
-        os.remove(filename)
-    with open(filename, "w") as file:
+    if os.path.exists(output_filename):
+        os.remove(output_filename)
+    with open(output_filename, "w") as file:
         json.dump(results, file, indent=4, default=str)
-        print(f"Saved results to {filename}")
+        print(f"Saved results to {output_filename}")
     return 1
-
-
-def get_list_of_aws_profiles(credentials_file):
-    """Get a list of profiles from the AWS Credentials file"""
-    config = configparser.RawConfigParser()
-    config.read(credentials_file)
-    sections = config.sections()
-    legitimate_sections = []
-    for section in sections:
-        # https://github.com/broamski/aws-mfa#credentials-file-setup
-        broamski_suffix = "-long-term"
-        # pylint: disable=no-else-continue
-        if section.endswith(broamski_suffix):
-            # skip it if it's not a real profile we want to evaluate
-            continue
-        else:
-            legitimate_sections.append(section)
-    return legitimate_sections
-
-
-def login(profile_name, service="iam"):
-    """Log in to AWS and return a boto3 session."""
-    default_region = os.environ.get("AWS_REGION", "us-east-1")
-    session_data = {"region_name": default_region}
-    if profile_name:
-        session_data["profile_name"] = profile_name
-
-    session = boto3.Session(**session_data)
-
-    # Return the service requested by the function - either sts or iam
-    if service:
-        this_session = session.client(service)
-    # By default return IAM
-    else:
-        this_session = session.client("iam")
-    return this_session
diff --git a/docs/user-guide/download.md b/docs/user-guide/download.md
@@ -1,37 +1,26 @@
-# Downloading Account Authorization Details
+#### Downloading Account Authorization Details
 
-The `download` command downloads a large JSON file containing all the AWS IAM information in your account. This is done via the [aws iam get-account-authorization-details](https://docs.aws.amazon.com/cli/latest/reference/iam/get-account-authorization-details.html) API call. It stores them in `account-alias.json`.
+We can scan an entire AWS account and generate reports. To do this, we leverage the AWS IAM [get-account-authorization-details](https://docs.aws.amazon.com/cli/latest/reference/iam/get-account-authorization-details.html) API call, which downloads a large JSON file (around 100KB per account) that contains all of the IAM details for the account. This includes data on users, groups, roles, customer-managed policies, and AWS-managed policies.
 
-The `scan` command requires that file.
+* You must have AWS credentials configured that can be used by the CLI.
 
-## Quick start
+* You must have the privileges to run [iam:GetAccountAuthorizationDetails](https://docs.aws.amazon.com/IAM/latest/APIReference/API_GetAccountAuthorizationDetails.html). The `arn:aws:iam::aws:policy/SecurityAudit` policy includes this, as do many others that allow Read access to the IAM Service.
 
-* Set your AWS access keys as environment variables:
+* To download the account authorization details, ensure you are authenticated to AWS, then run `cloudsplaining`'s `download` command:
 
 ```bash
-export AWS_ACCESS_KEY_ID=...
-export AWS_SECRET_ACCESS_KEY=...
-# If you are using MFA or STS; optional but highly recommended
-export AWS_SESSION_TOKEN=...
+cloudsplaining download
 ```
 
-* Download the account authorization details
+* If you prefer to use your `~/.aws/credentials` file instead of environment variables, you can specify the profile name:
 
 ```bash
-cloudsplaining download
+cloudsplaining download --profile myprofile
 ```
 
-## Additional Details
-
-#### Order of Precedence
-
-* **Environment variables**: The `download` command will first look for the existence of your AWS access keys in environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`).
-  - Note: If you do not have AWS_SESSION_TOKEN set and are using static access keys, I highly recommend the use of [aws-mfa](https://github.com/broamski/aws-mfa) for security reasons.
-
-* **Shared Credentials file**:
-  - If those environment variables are not set, it will then use the `default` profile in your `~/.aws/credentials` file, if a different profile name is not provided via the argument `--profile`.
-  - If you specify `--profile all`, it will run the download command recursively for every profile in your `~/.aws/credentials` file.
+It will download a JSON file in your current directory that contains your account authorization detail information.
 
+## Additional Details
 
 ### Required AWS IAM Policy
 

diff --git a/docs/user-guide/scan-account.md b/docs/user-guide/scan-account.md
@@ -1,14 +1,40 @@
-# Scanning an Account
+#### Scanning the Authorization Details file
 
-Scan the Account Authorization details file with the following command
+Now that we've downloaded the account authorization file, we can scan *all* of the AWS IAM policies with `cloudsplaining`.
 
+Run the following command:
+
+```bash
+cloudsplaining scan --exclusions-file exclusions.yml --input examples/files/example.json --output examples/files/
 ```
-cloudsplaining scan --input default.json --exclusions-file my-exclusions.yml
-```
 
-* It will generate three files:
-  1. The single-file HTML report
-  2. The triage CSV worksheet, and
-  3. The raw JSON data file
+It will create an HTML report like [this](https://opensource.salesforce.com/cloudsplaining/):
+
+> ![](docs/_images/cloudsplaining-report.gif)
 
 
+It will also create a raw JSON data file:
+
+* `default-iam-results.json`: This contains the raw JSON output of the report. You can use this data file for operating on the scan results for various purposes. For example, you could write a Python script that parses this data and opens up automated JIRA issues or Salesforce Work Items. An example entry is shown below. The full example can be viewed at [examples/output/example-authz-details-results.json](examples/files/iam-results-example.json)
+
+```json
+{
+    "example-authz-details": [
+        {
+            "AccountID": "012345678901",
+            "ManagedBy": "Customer",
+            "PolicyName": "InsecureUserPolicy",
+            "Arn": "arn:aws:iam::012345678901:user/userwithlotsofpermissions",
+            "ActionsCount": 2,
+            "ServicesCount": 1,
+            "Actions": [
+                "s3:PutObject",
+                "s3:PutObjectAcl"
+            ],
+            "Services": [
+                "s3"
+            ]
+        }
+    ]
+}
+```
diff --git a/docs/user-guide/troubleshooting.md b/docs/user-guide/troubleshooting.md
@@ -1,14 +1,13 @@
 # Troubleshooting
 
-### Running the command (Path issues)
-
-* *I followed the installation instructions but can't execute the program via command line. What do I do?*
+**I followed the installation instructions but can't execute the program via command line at all. What do I do?**
 
 This is likely an issue with your PATH. Your PATH environment variable is not considering the binary packages installed by `pip3`. On a Mac, you can likely fix this by entering the command below, depending on the versions you have installed. YMMV.
 
 ```bash
-# Python 3.7
 export PATH=$HOME/Library/Python/3.7/bin/:$PATH
-# Python 3.8
-export PATH=$HOME/Library/Python/3.8/bin/:$PATH
 ```
+
+**I followed the installation instructions but I am receiving a `ModuleNotFoundError` that says `No module named policy_sentry.analysis.expand`. What should I do?**
+
+Try upgrading to the latest version of Cloudsplaining. This error was fixed in version 0.0.10.
diff --git a/setup.py b/setup.py
@@ -37,7 +37,7 @@ def get_version():
     packages=setuptools.find_packages(exclude=['test*', 'tmp*']),
     tests_require=TESTS_REQUIRE,
     install_requires=[
-        'policy_sentry',
+        'policy_sentry>=0.8.0.3',
         'click',
         'click_log',
         'schema',