Skip to content
This repository has been archived by the owner on Aug 1, 2024. It is now read-only.

Latest commit

 

History

History
557 lines (478 loc) · 20.8 KB

install.md

File metadata and controls

557 lines (478 loc) · 20.8 KB
Raw

AWS CloudFormation Template Sync Controller for Flux installation guide

This document covers the installation of the AWS CloudFormation Template Sync Controller for Flux into your Kubernetes cluster.

You can find instructions for running the CloudFormation controller on a local Kubernetes cluster in the development guide.

  1. Prerequisites
  2. Create an AWS IAM policy
  3. Create AWS credentials
  4. Register the CloudFormation controller repository with Flux
  5. Deploy the CloudFormation controller
    1. Use IAM roles for service accounts on an Amazon EKS cluster (recommended)
    2. Use AWS credentials in environment variables
    3. Use AWS credentials in a mounted file
  6. Validate the CloudFormation controller deployment
  7. Enable Flux notifications for the CloudFormation controller
  8. Security recommendations
    1. Kubernetes cluster security
    2. Kubernetes user permissions

Prerequisites

These instructions assume you already have a Kubernetes cluster with Flux installed. See the Flux documentation for instructions on installing Flux into your Kubernetes cluster. Note that the CloudFormation controller requires the Flux source controller v0.18.0 or newer.

These instructions also assume that you already created the following prerequisite resources. You can use the sample CloudFormation template for creating these resources.

  • Flux configuration repository: These instructions assume that Flux is configured to manage itself from a Git repository, for example using the flux bootstrap command.
  • At least one CloudFormation template repository: Through the Flux source controller, the CloudFormation controller can deploy CloudFormation templates stored in a Git repository such as an AWS CodeCommit repository, in a bucket such as an Amazon S3 bucket, or in an OCI repository such as an Amazon ECR repository. See the Flux documentation for a guide to various approaches for organizing the repositories that store your CloudFormation templates.
  • Amazon S3 bucket: The CloudFormation controller requires an S3 bucket. When the controller syncs a CloudFormation template into a CloudFormation stack in your AWS account, it will first upload the template to S3 and then provide a link to the S3 object to CloudFormation. To minimize storage costs, you can safely enable a lifecycle rule on the S3 bucket to expire objects after one day.

Create an AWS IAM policy

The CloudFormation controller needs an IAM policy that allows it to deploy CloudFormation stacks in your AWS account. For example, you can create an IAM policy using the AWS CLI.

$ aws iam create-policy --policy-name my-policy --policy-document file://my-policy.json

We recommend that the AWS credentials used by the CloudFormation controller have the least privileged permissions needed to deploy your CloudFormation stacks in your AWS account.

The CloudFormation controller requires the following IAM permissions for managing CloudFormation stacks in your AWS account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudformation:ContinueUpdateRollback",
        "cloudformation:CreateChangeSet",
        "cloudformation:DeleteChangeSet",
        "cloudformation:DeleteStack",
        "cloudformation:DescribeChangeSet",
        "cloudformation:DescribeStacks",
        "cloudformation:ExecuteChangeSet"
      ],
      "Resource": [
        "arn:aws:cloudformation:us-west-2:123456789012:stack/*"
      ]
    }
  ]
}

The CloudFormation controller also requires the following IAM permissions to upload your CloudFormation templates to your S3 bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::<your S3 bucket name>/flux-*.template"
      ]
    }
  ]
}

The CloudFormation controller also requires permissions on behalf of CloudFormation to download your CloudFormation templates from your S3 bucket and to provision the resources defined in your CloudFormation templates.

For example, if your CloudFormation templates define AWS::DynamoDB::Table resources, the CloudFormation controller may need the following permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::<your S3 bucket name>/flux-*.template"
      ],
      "Condition": {
        "ForAnyValue:StringEquals": {
          "aws:CalledVia": [
            "cloudformation.amazonaws.com"
          ]
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:CreateTable",
        "dynamodb:DeleteTable",
        "dynamodb:DescribeTable",
        "dynamodb:DescribeTimeToLive",
        "dynamodb:UpdateTimeToLive",
        "dynamodb:UpdateContributorInsights",
        "dynamodb:UpdateContinuousBackups",
        "dynamodb:DescribeContinuousBackups",
        "dynamodb:DescribeContributorInsights",
        "dynamodb:EnableKinesisStreamingDestination",
        "dynamodb:DisableKinesisStreamingDestination",
        "dynamodb:DescribeKinesisStreamingDestination",
        "dynamodb:ListTagsOfResource",
        "dynamodb:TagResource",
        "dynamodb:UntagResource",
        "dynamodb:UpdateTable",
      ],
      "Resource": "arn:aws:dynamodb:us-west-2:123456789012:table/*",
      "Condition": {
        "ForAnyValue:StringEquals": {
          "aws:CalledVia": [
            "cloudformation.amazonaws.com"
          ]
        }
      }
    }
  ]
}

Create AWS credentials

The CloudFormation controller relies on the default behavior of the AWS SDK for Go V2 to determine the AWS credentials that it uses to authenticate with AWS APIs.

The CloudFormation controller searches for credentials in the following order:

  1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
  2. Web identity token credentials (including running in an Amazon EKS cluster using IAM roles for service accounts)
  3. Shared credentials and config ini files (~/.aws/credentials, ~/.aws/config)
  4. Amazon Elastic Container Service (Amazon ECS) task metadata service
  5. Amazon Elastic Compute Cloud (Amazon EC2) instance metadata service

As a best practice, we recommend that you use short lived AWS credentials for the CloudFormation controller, for example using IAM roles for service accounts on an EKS cluster.

We also recommend that the AWS credentials used by the CloudFormation controller have the least privileged permissions needed to deploy your CloudFormation stacks in your AWS account. See the AWS IAM permissions section below for example policies to attach to your IAM user or role.

For example, the following commands use eksctl to create an IAM OIDC identity provider for your EKS cluster, create an IAM role, attach an IAM policy to the IAM role, and associate the IAM role with a Kubernetes service account for the CloudFormation controller in your EKS cluster's flux-system namespace.

$ eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve

$ eksctl create iamserviceaccount \
    --cluster my-cluster \
    --namespace flux-system \
    --name cfn-controller \
    --role-only \
    --role-name "AWSCloudFormationControllerFluxIRSARole" \
    --attach-policy-arn arn:aws:iam::123456789012:policy/my-policy \
    --approve

Register the CloudFormation controller repository with Flux

In your Flux configuration repository, create a file named cfn-controller-source.yaml with the configuration below to register this GitHub repository with your Flux installation.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: aws-cloudformation-controller-for-flux
  namespace: flux-system
spec:
  interval: 1h
  timeout: 60s
  ref:
    branch: main
  url: https://github.com/awslabs/aws-cloudformation-controller-for-flux

Update the GitRepository .spec.ref field to point to the CloudFormation controller release version or commit ID you want to deploy. For example, replace branch: main with tag: v0.0.1 to pin the installation to CloudFormation controller version 0.0.1.

Commit and push the cfn-controller-source.yaml file to your Flux configuration repository.

Run the following commands to verify that Flux can successfully connect to this repository.

$ flux reconcile source git flux-system
$ flux reconcile source git aws-cloudformation-controller-for-flux

Deploy the CloudFormation controller using Flux

In your Flux configuration repository, create a file named cfn-controller.yaml, fill in the appropriate contents, then commit and push the file to your Flux configuration repository.

The contents of the cfn-controller.yaml file depends on how you choose to provide AWS credentials to the CloudFormation controller; see the three options below. In all cases, you will need to know the name of the AWS region where the CloudFormation controller should deploy CloudFormation stacks and the name of the S3 bucket where the CloudFormation controller should upload CloudFormation templates.

Option 1: Short-lived credentials using IAM roles for service accounts on an EKS cluster (recommended)

Copy and paste the following configuration into the cfn-controller.yaml file. Update the eks.amazonaws.com/role-arn value with the correct IAM role ARN. Update the AWS_REGION and TEMPLATE_BUCKET values with the correct values for your region and S3 bucket.

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: aws-cloudformation-controller-for-flux
  namespace: flux-system
spec:
  interval: 5m
  path: ./config/default
  prune: true
  wait: true
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: aws-cloudformation-controller-for-flux
  patches:
    - patch: |
        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: cfn-controller
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/AWSCloudFormationControllerFluxIRSARole
      target:
        kind: ServiceAccount
        name: cfn-controller
    - patch: |
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: cfn-controller
        spec:
          template:
            spec:
              containers:
              - name: manager
                env:
                  - name: AWS_REGION
                    value: "us-west-2"
                  - name: TEMPLATE_BUCKET
                    value: "my-cloudformation-templates-bucket"
      target:
        kind: Deployment
        name: cfn-controller

Option 2: Long-term credentials as environment variables

Create a Kubernetes secret that contains your IAM user's credentials. First, encode your credentials:

$ echo -n 'FAKEAWSACCESSKEYID' | base64 -w 0
RkFLRUFXU0FDQ0VTU0tFWUlE

$ echo -n 'FAKEAWSSECRETACCESSKEY' | base64 -w 0
RkFLRUFXU1NFQ1JFVEFDQ0VTU0tFWQ==

Then create a file named secret.yaml containing the Kubernetes secret configuration:

apiVersion: v1
kind: Secret
metadata:
  name: aws-creds-for-cfn-controller
  namespace: flux-system
type: Opaque
data:
  AWS_ACCESS_KEY_ID: RkFLRUFXU0FDQ0VTU0tFWUlE
  AWS_SECRET_ACCESS_KEY: RkFLRUFXU1NFQ1JFVEFDQ0VTU0tFWQ==

Apply the secret to your Kubernetes cluster, then delete the configuration file.

$ kubectl apply -f secret.yaml
$ rm secret.yaml

Copy and paste the following configuration into the cfn-controller.yaml file. Update the AWS_REGION and TEMPLATE_BUCKET values with the correct values for your region and S3 bucket.

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: aws-cloudformation-controller-for-flux
  namespace: flux-system
spec:
  interval: 5m
  path: ./config/default
  prune: true
  wait: true
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: aws-cloudformation-controller-for-flux
  patches:
    - patch: |
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: cfn-controller
        spec:
          template:
            spec:
              containers:
              - name: manager
                env:
                  - name: AWS_REGION
                    value: "us-west-2"
                  - name: TEMPLATE_BUCKET
                    value: "my-cloudformation-templates-bucket"
                  - name: AWS_ACCESS_KEY_ID
                    valueFrom:
                      secretKeyRef:
                        name: aws-creds-for-cfn-controller
                        key: AWS_ACCESS_KEY_ID
                  - name: AWS_SECRET_ACCESS_KEY
                    valueFrom:
                      secretKeyRef:
                        name: aws-creds-for-cfn-controller
                        key: AWS_SECRET_ACCESS_KEY
      target:
        kind: Deployment
        name: cfn-controller

Option 3: Long-term credentials in a mounted credentials file

Create a file named credentials that contains your IAM user's credentials:

[default]
aws_access_key_id=FAKEAWSACCESSKEYID
aws_secret_access_key=FAKEAWSSECRETACCESSKEY

Create a Kubernetes secret that contains credentials file, then delete the credentials file.

$ kubectl create secret generic aws-creds-for-cfn-controller -n flux-system --from-file ./credentials
$ rm ./credentials

Copy and paste the following configuration into the cfn-controller.yaml file. Update the AWS_REGION and TEMPLATE_BUCKET values with the correct values for your region and S3 bucket.

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: aws-cloudformation-controller-for-flux
  namespace: flux-system
spec:
  interval: 5m
  path: ./config/default
  prune: true
  wait: true
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: aws-cloudformation-controller-for-flux
  patches:
    - patch: |
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: cfn-controller
        spec:
          template:
            spec:
              containers:
              - name: manager
                env:
                  - name: AWS_REGION
                    value: "us-west-2"
                  - name: TEMPLATE_BUCKET
                    value: "my-cloudformation-templates-bucket"
                volumeMounts:
                - name: aws-creds
                  mountPath: "/.aws"
                  readOnly: true
              volumes:
              - name: aws-creds
                secret:
                  secretName: aws-creds-for-cfn-controller
      target:
        kind: Deployment
        name: cfn-controller

Configure CloudFormation controller flags

If needed, you can configure the CloudFormation controller using command line flags. For the full set of available flags, run:

$ docker run --rm public.ecr.aws/aws-cloudformation/aws-cloudformation-controller-for-flux --help

Specify the command line flags you want to use in your cfn-controller.yaml file, then commit and push the changes to your Flux configuration repository. For example, the following Kustomize patch added to the cfn-controller.yaml file will increase the number of concurrent CloudFormationStack reconciles to 10 using the --controller flag.

  patches:
    - patch: |
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: cfn-controller
        spec:
          template:
            spec:
              containers:
              - name: manager
                args: ["--concurrent", "10"]

Validate the CloudFormation controller deployment

Validate that Flux is able to successfully deploy the CloudFormation controller configuration:

$ flux reconcile source git flux-system

$ flux reconcile kustomization aws-cloudformation-controller-for-flux

$ kubectl rollout status deployment/cfn-controller -n flux-system

$ kubectl logs deployment/cfn-controller -n flux-system

Enable Flux notifications for the CloudFormation controller

Notifications must be enabled for third-party Flux controllers, including the CloudFormation controller. See the Flux documentation for more details.

In your Flux configuration repository, update the flux-system/kustomization.yaml file to match the following content. Commit and push the changes to your Flux configuration repository.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - patch: |
      - op: add
        path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/eventSources/items/properties/kind/enum/-
        value: CloudFormationStack
      - op: add
        path: /spec/versions/1/schema/openAPIV3Schema/properties/spec/properties/eventSources/items/properties/kind/enum/-
        value: CloudFormationStack
    target:
      kind: CustomResourceDefinition
      name:  alerts.notification.toolkit.fluxcd.io
  - patch: |
      - op: add
        path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/resources/items/properties/kind/enum/-
        value: CloudFormationStack
      - op: add
        path: /spec/versions/1/schema/openAPIV3Schema/properties/spec/properties/resources/items/properties/kind/enum/-
        value: CloudFormationStack
    target:
      kind: CustomResourceDefinition
      name:  receivers.notification.toolkit.fluxcd.io
  - patch: |
      - op: add
        path: /rules/-
        value:
          apiGroups: [ 'cloudformation.contrib.fluxcd.io' ]
          resources: [ '*' ]
          verbs: [ '*' ]
    target:
      kind: ClusterRole
      name:  crd-controller-flux-system

Security recommendations

Kubernetes cluster security

We recommend following all security best practices defined by the Flux project when configuring your Flux components and the CloudFormation controller. We also recommend following the Flux project's additional security best practices for shared cluster multi-tenancy, including node isolation and network isolation for your Flux components and the CloudFormation controller.

For information on how to achieve node isolation and network isolation for your Flux components and the CloudFormation controller on Amazon EKS clusters, see the following EKS Best Practices Guides:

We recommend enabling mTLS encryption and authentication for communication between Flux components, for example between the CloudFormation controller and the Flux source controller. A service mesh can be used to enable mTLS, such as AWS App Mesh, and Flux may support mTLS natively in the future (see discussion here).

Kubernetes user permissions

We recommend that users with access to your Kubernetes cluster have the least privileged permissions needed for interacting with the CloudFormation controller. We have provided two sample Kubernetes roles that can be used to grant permissions to your users.