ArgoCD + Terraform + Cognito - Challenges,Issues, and Ideas

[renanmb]

Following from Slack as Apoorva Kulkarni suggested (https://kubeflow.slack.com/archives/CKBA5D0MU/p1679939808948949?thread_ts=1679847948.456869&cid=CKBA5D0MU)
cc: @Isaac Mosquera (AWS), @Alex Lines

While exploring I came across challenges with Gitops while trying to use Terraform + ArgoCD.
The reasoning is to work around existing issues.

• Terraform + Cognito has a long lasting issue that we cannot update throught Terraform, see below:

  • Terraform Cognito User Pool can't run update

Also, even though Terraform is awesome as IaC for certain resources, it can be terrible for others.
Therefore, Terraform + ArgoCD has a beautiful harmony. Article: Terraform and ArgoCD in beautiful harmony

• Terraform to manage Cloud Provider services (Network, DNS, Compute, etc).
• ArgoCD to manage Kubernetes deployments (EKS addons is a start), typically packaged as different Helm charts, with dynamic properties defined as Helm values.

Other challenge has been managing the ASG (scale up or scale down events from zero)
Even though it is possible to set CloudWatch alarm for a scale up or scale down event in the ASG using the Step Scale Policies. We can only scale up from 0 running instances using the current metrics.
The challenge is exposing iddleness from within kubeflow so it can scale down to zero instances.

Why this matters ?
Automated resource utilization.

In Summary:

• Need a work around Terraform Cognito User Pool using IaC.
• Need ArgoCD to manage certain things.
• Need a way to extract a metric from kubeflow for iddleness so we can apply to ASG Scale Policies.

[infa-amchandr]

+1