Skip to content

Latest commit

 

History

History
380 lines (310 loc) · 16.9 KB

aws-cloud-integrations.md

File metadata and controls

380 lines (310 loc) · 16.9 KB

Kubecost pulls asset prices from the public AWS pricing API by default. To have accurate pricing information from AWS, you can integrate directly with your account. This integration will properly account for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, spot usage and more. This resource describes the required steps for achieving this.

Cost and Usage Report Integration

Step 1: Setting up the CUR

https://docs.aws.amazon.com/cur/latest/userguide/cur-ate-setup.html#create-athena-cur

Note the name of the bucket you create for CUR data. This will be used in following steps.

Step 2: Setting up IAM permissions

Add via Cloudformation:

Kubecost offers a set of cloudformation templates to help set your IAM roles up. If you’re new to provisioning IAM roles, we suggest downloading our templates and using the cloudformation wizard to set these up: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html . Download template files from the URLs provided below and upload them as the stack template in the Creating a stack > Selecting a stack template step.

My kubernetes clusters all run in the same account as the master payer account.

  • Download this file: https://raw.githubusercontent.com/kubecost/cloudformation/master/kubecost-single-account-permissions.yaml

  • Navigate to https://console.aws.amazon.com/cloudformation

  • Choose Create New Stack if you have never used AWS CloudFormation before. Otherwise, choose Create Stack.

  • Under Prepare template, choose Template is ready.

  • Under Template source, choose Upload a template file.

  • Select Choose file.

  • Choose the downloaded .yaml template, and then choose Open.

  • Choose Next.

  • For Stack name, enter a name for your template

  • Set the following parameters:

    • AthenaCURBucket: The bucket where the CUR is sent from the “Setting up the CUR” step
    • SpotDataFeedBucketName: Optional. The bucket where the spot data feed is sent from the “Setting up the Spot Data feed” step (see below)

  • Choose Next.

  • Choose Next

  • At the bottom of the page, select I acknowledge that AWS CloudFormation might create IAM resources.

  • Choose Create Stack

My kubernetes clusters run in different accounts from the master payer account
  • On each sub account running kubecost
    • Download this file: https://raw.githubusercontent.com/kubecost/cloudformation/master/kubecost-sub-account-permissions.yaml
    • Navigate to https://console.aws.amazon.com/cloudformation
    • Choose Create New Stack if you have never used AWS CloudFormation before. Otherwise, choose Create Stack.
    • Under Prepare template, choose Template is ready.
    • Under Template source, choose Upload a template file.
    • Select Choose file.
    • Choose the downloaded .yaml template, and then choose Open.
    • Choose Next.
    • For Stack name, enter a name for your template
    • Set the following parameters:
      • MasterPayerAccountID: The account ID of the master payer account where the CUR has been created
      • SpotDataFeedBucketName: The bucket where the spot data feed is sent from the “Setting up the Spot Data feed” step
    • Choose Next.
    • Choose Next
    • At the bottom of the page, select I acknowledge that AWS CloudFormation might create IAM resources.
    • Choose Create Stack
  • On the master payer account

Add manually

My Kubernetes clusters run in the same account as the master payer account

Attach both of the following policies to the same role or user. Use a user if you intend to integrate via servicekey, and a role if via IAM annotation (See more below under Via Pod Annotation by EKS). The SpotDataAccess policy statment is optional if the spot data feed is configured (see “Setting up the Spot Data feed” step below)

        {
           "Version": "2012-10-17",
           "Statement": [
              {
                 "Sid": "AthenaAccess",
                 "Effect": "Allow",
                 "Action": [
                    "athena:*"
                 ],
                 "Resource": [
                    "*"
                 ]
              },
              {
                 "Sid": "ReadAccessToAthenaCurDataViaGlue",
                 "Effect": "Allow",
                 "Action": [
                    "glue:GetDatabase*",
                    "glue:GetTable*",
                    "glue:GetPartition*",
                    "glue:GetUserDefinedFunction",
                    "glue:BatchGetPartition"
                 ],
                 "Resource": [
                    "arn:aws:glue:*:*:catalog",
                    "arn:aws:glue:*:*:database/athenacurcfn*",
                    "arn:aws:glue:*:*:table/athenacurcfn*/*"
                 ]
              },
              {
                 "Sid": "AthenaQueryResultsOutput",
                 "Effect": "Allow",
                 "Action": [
                    "s3:GetBucketLocation",
                    "s3:GetObject",
                    "s3:ListBucket",
                    "s3:ListBucketMultipartUploads",
                    "s3:ListMultipartUploadParts",
                    "s3:AbortMultipartUpload",
                    "s3:CreateBucket",
                    "s3:PutObject"
                 ],
                 "Resource": [
                    "arn:aws:s3:::aws-athena-query-results-*"
                 ]
              },
	   
	      
              {
                 "Sid": "S3ReadAccessToAwsBillingData",
                 "Effect": "Allow",
                 "Action": [
                    "s3:Get*",
                    "s3:List*"
                 ],
                 "Resource": [
                    "arn:aws:s3:::${AthenaCURBucket}*"
                 ]
              }
           ]
        }
	{
           "Version": "2012-10-17",
           "Statement": [
              {
                 "Sid": "SpotDataAccess",
                 "Effect": "Allow",
                 "Action": [
                    "s3:ListAllMyBuckets",
                    "s3:ListBucket",
                    "s3:HeadBucket",
                    "s3:HeadObject",
                    "s3:List*",
                    "s3:Get*"
                 ],
                 "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
              }
           ]
        }
My Kubernetes clusters run in different accounts

On each sub account running kubecost, attach both of the following policies to the same role or user. Use a user if you intend to integrate via servicekey, and a role if via IAM annotation (See more below under Via Pod Annotation by EKS). The SpotDataAccess policy statment is optional if the spot data feed is configured (see “Setting up the Spot Data feed” step below)

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Sid": "AssumeRoleInMasterPayer",
                     "Effect": "Allow",
                     "Action": "sts:AssumeRole",
                     "Resource": "arn:aws:iam::${MasterPayerAccountID}:role/KubecostRole-${This-account’s-id}"
                  }
               ]
	}

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Sid": "SpotDataAccess",
                     "Effect": "Allow",
                     "Action": [
                        "s3:ListAllMyBuckets",
                        "s3:ListBucket",
                        "s3:HeadBucket",
                        "s3:HeadObject",
                        "s3:List*",
                        "s3:Get*"
                     ],
                     "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
                  }
               ]
	}

On the masterpayer account, attach this policy to a role:

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Sid": "AthenaAccess",
                     "Effect": "Allow",
                     "Action": [
                        "athena:*"
                     ],
                     "Resource": [
                        "*"
                     ]
	},
	{
                     "Sid": "ReadAccessToAthenaCurDataViaGlue",
                     "Effect": "Allow",
                     "Action": [
                        "glue:GetDatabase*",
                        "glue:GetTable*",
                        "glue:GetPartition*",
                        "glue:GetUserDefinedFunction",
                        "glue:BatchGetPartition"
                     ],
                     "Resource": [
                        "arn:aws:glue:*:*:catalog",
                        "arn:aws:glue:*:*:database/athenacurcfn*",
                        "arn:aws:glue:*:*:table/athenacurcfn*/*"
                     ]
                  },
                  {
                     "Sid": "AthenaQueryResultsOutput",
                     "Effect": "Allow",
                     "Action": [
                        "s3:GetBucketLocation"
                        "s3:GetObject",
                        "s3:ListBucket",
                        "s3:ListBucketMultipartUploads",
                        "s3:ListMultipartUploadParts",
                        "s3:AbortMultipartUpload",
                        "s3:CreateBucket",
                        "s3:PutObject,
                     ],
                     "Resource": [
                        "arn:aws:s3:::aws-athena-query-results-*"
                     ]
                  },
                  {
                     "Sid": "S3ReadAccessToAwsBillingData",
                     "Effect": "Allow",
                     "Action": [
                        "s3:Get*",
                        "s3:List*"
                     ],
                     "Resource": [
                        "arn:aws:s3:::${AthenaCURBucket}*"
                     ]
                  }
               ]
	}

You will then need to add the following trust statement to the role the policy is attached to:

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Effect": "Allow",
                     "Principal": {
                        "AWS": `'arn:aws:iam::${KubecostClusterID}:root'`
                     },
                     "Action": [
                        "sts:AssumeRole"
                     ]
                  }
               ]
            }

Step 3: Attaching IAM permissions to Kubecost

Now that the policies have been created, we will need to attach those policies to Kubecost. We support the following methods:

Attach via Service Key And Kubernetes Secret

  • Navigate to https://console.aws.amazon.com/iam Access Management > Users . Find the Kubecost User and select Security Credentials > Create Access Key. Note the Access key ID and Secret access key. You'll use it to either Create a secret from helm values or Create and use an existing secret.

    Create a secret from helm values
    • Set .Values.kubecostProductConfigs.awsServiceKeyName to Access key ID
    • Set .Values.kubecostProductConfigs.awsServiceKeyPassword to Secret access key
    • Note that this will leave your secrets unencrypted in values.yaml. Use an existing secret as in the next step to avoid this.
    Create and use an existing secret

    If you commit your helm values to source control, you may want to create a secret in a different way and import that secret to kubecost.

    • Create a json file named service-key.json of the following format 
               {
         	"`aws_access_key_id": <ACCESS_KEY_ID>,`
         	"aws_secret_access_key": <ACCESS_KEY_SECRET>
         }
    • Create a secret from file in the namespace kubecost is deployed in: kubectl create secret generic <name> --from-file=service-key.json --namespace <kubecost>
    • Set .Values.kubecostProductConfigs.serviceKeySecretName to the name of this secet. Note also that .Values.kubecostProductConfigs.awsServiceKeyName and .Values.kubecostProductConfigs.awsServiceKeyPassword should be unset if adding the service key from values this way.
Attach via Service Key on Kubecost frontend
  • Navigate to https://console.aws.amazon.com/iam Access Management > Users . Find the Kubecost User and select Security Credentials > Create Access Key. Note the Access key ID and Secret access key.
  • You can add the Access key ID and Secret access key on /settings.html > External Cloud Cost Configuration (AWS) > Update and setting Service key name to Access key ID and Service key secret to Secret access key
Attach via Pod Annotation on EKS

Step 4: Provide CUR config values to Kubecost

These values can either be set from the kubecost frontend or via .Values.kubecostProductConfigs in the helm chart. Note that if you set any kubecostProductConfigs from the helm chart, all changes via the frontend will be overridden on pod restart.

  • athenaProjectID e.g. "530337586277" # The AWS AccountID where the Athena CUR is.

  • athenaBucketName A result bucket you’ve created that kubecost has permission to access, of the form aws-athena-query-results-<your-bucket-name>

  • athenaRegion The aws region athena is running in

  • athenaDatabase the name of the database created by the CUR setup

  • athenaTable the name of the table created by the CUR setup

  • If you are using a multi-account setup, you will also need to set .Values.kubecostProductConfigs.masterPayerARN To the arn of the role in the masterpayer account, e.g. arn:aws:iam::530337586275:role/KubecostRole.

Want to relate out-of-cluster costs to k8s resources via tags?

Spot Data feed integration

Kubecost will reconcile your spot prices with CUR billing reports as they become available (usually 1-2 days), but pricing data can be pulled hourly by integrating directly with the AWS spot feed. To enable, follow these steps:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-data-feeds.html

Note the name of the bucket you create for spot data. This will be used in the following step.

Configuring the Spot Data Feed in Kubecost

These values can either be set from the kubecost frontend or via .Values.kubecostProductConfigs in the helm chart. Note that if you set any kubecostProductConfigs from the helm chart, all changes via the frontend will be deleted on pod restart

awsSpotDataRegion region of your spot data bucket

awsSpotDataBucket the configured bucket for the spot data feed

awsSpotDataPrefix optional configured prefix for your spot data feed bucket