Upgrade eks-core-services in CircleCI

[nadiia-kotelnikova]

Hello all,

i ran into a problem during the EKS cluster upgrade: we recently deployed ECS deploy runner and have not yet experienced with it. When I upgraded the eks-core-service module, the CircleCI pipeline failed with these errors:

[ecs-deploy-runner][2023-01-16T16:42:43+0000] ╷
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │ Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │ 
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │   with module.alb_ingress_controller["enable"].helm_release.aws_alb_ingress_controller,
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │   on .terraform/modules/alb_ingress_controller/modules/eks-alb-ingress-controller/main.tf line 48, in resource "helm_release" "aws_alb_ingress_controller":
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │   48: resource "helm_release" "aws_alb_ingress_controller" {
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │ 
[ecs-deploy-runner][2023-01-16T16:42:43+0000] ╵
[ecs-deploy-runner][2023-01-16T16:42:43+0000] ╷
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │ Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │ 
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │   with module.aws_for_fluent_bit["enable"].helm_release.aws_for_fluent_bit,
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │   on .terraform/modules/aws_for_fluent_bit/modules/eks-container-logs/main.tf line 48, in resource "helm_release" "aws_for_fluent_bit":
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │   48: resource "helm_release" "aws_for_fluent_bit" {
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │ 
[ecs-deploy-runner][2023-01-16T16:42:43+0000] ╵
[ecs-deploy-runner][2023-01-16T16:42:43+0000] ╷
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │ Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │ 
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │   with module.k8s_external_dns["enable"].helm_release.k8s_external_dns,
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │   on .terraform/modules/k8s_external_dns/modules/eks-k8s-external-dns/main.tf line 54, in resource "helm_release" "k8s_external_dns":
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │   54: resource "helm_release" "k8s_external_dns" {
[ecs-deploy-runner][2023-01-16T16:42:43+0000] │ 
[ecs-deploy-runner][2023-01-16T16:42:43+0000] ╵

My understanding is that ecs-deploy-runner ECS task does not perform Kubernetes authentication and does not have Kubernetes configuration. Does anybody know how to workaround this?

---

Tracked in ticket #109797

[josh-padnick]

@nadiia-caspar Thanks for your message. I've routed your question to an EKS expert here at Gruntwork, and we'll get back to you shortly.

That being said, it looks like the common Terraform resource causing the failure is helm_release, which is a Helm Chart, which probably does need to hit the K8s API. Unfortunately, I'm not familiar with how we currently configure the ECS Deploy Runner to authenticate to K8s in this situation.

[nadiia-kotelnikova]

thank you @josh-padnick. I am looking forward to the answer from the EKS expert:)

[nadiia-kotelnikova]

@josh-padnick could you please add s:CI/Pipelines label?

[josh-padnick]

Both requests have been fulfilled!

[autero1]

Without knowing the full details of your configuration, I'll try my best to explain...

For the ecs-deploy-runner to be able to interact with the EKS cluster, the IAM Role the runner uses, must be mapped in the aws-auth ConfigMap. Had the cluster been created with the IAM Role ecs-deploy-runner is using, this would be unnecessary, as EKS implicitly grants admin RBAC for the IAM role that the cluster was created with. I'm assuming the cluster was created with a different role?

To fix the issue, the ECS Deploy Runner IAM Role has to be added to aws-auth ConfigMap. If you're using the eks-aws-auth-merger, you can use the eks-k8s-role-mapping to create an entry in the aws-auth ConfigMap, e.g.

module "ecs_deploy_runner_eks_k8s_role_mapping" {
  source = "git::[email protected]:gruntwork-io/terraform-aws-eks.git//modules/eks-k8s-role-mapping?ref=v0.x.x"

  name      = "ecs-deploy-runner"
  namespace = "whatever-namespace-you-use-for-auth-merger"

  eks_worker_iam_role_arns                   = []
  eks_fargate_profile_executor_iam_role_arns = []

  iam_role_to_rbac_group_mappings = {
    # I'm assuming you want admin level permissions in the cluster, because you'll be deploying
    # RBAC resources, hence the system:masters
    "your-ecs-deploy-runner-iam-role"    = ["system:masters"]
  }

  config_map_labels = {
    eks-cluster = module.eks_cluster.eks_cluster_name
  }
}

Make sure you're not overwriting the entire aws-auth ConfigMap 😅 and check the plan results carefully before applying. Note that you'll have to deploy the module with an IAM Role that has sufficient permissions in the EKS cluster. After the aws-auth ConfigMap has been updated, applying with the ecs-deploy-runner should work.

Hope this helps!

[nadiia-kotelnikova]

Awesome @autero1. Thank you very much. I'll give it a try!