Digit v2.6 quick start deployment on AWS cloud (hurdles, solutions & suggestions)

[subhashini-egov]

Followed this link – https://docs.digit.org/v/v2.6/setup/quickstart/1.-infra-setup (on a Cloud - AWS) – to deploy Digit on the AWS eGov cloud. Below are some suggestions/observations:

1. In general, having an installer or UI wizard check the hardware requirements of the machine before proceeding with installation will be good. Even a simple script that outputs “Yes” or “No, don’t proceed” will be a good starting point.

2. No terraform 0.14.10 on MacOSX. Closest is 0.13 but that is not supported with the Apple M1 chip. Latest version of Terraform (v 1.2) got installed via Homebrew and worked well.

3. https://docs.digit.org/v/v2.6/setup/install-on-cloud/on-aws/3.-setup-aws-account -- gives instructions on how to set up AWS account. Uses a default profile name and creates AWS credentials under ~/.aws/credentials using the default profile name. Further down, Step 5 asks the user to change the profile names (from default to something else) in the variables.tf file. If the profile name doesn't match what's in the .aws/credentials file, we run into 403 and 500 errors. Solution: Create a separate section in the ~/.aws/credentials file with the new profile name created in Step 5 and copy the access token/secret key etc..there as well.
   Make it explicit that session tokens from AWS will expire and if someone is setting up over a period of 24 hours with breaks, these will have to be renewed on the command line often before running other commands.

4. For the below step, move the note on how to get the private IP of VM right next to the command as that is visually somehow getting "distanced" or "lost".
   sudo k3d cluster create
   --api-port <VM_Private_IP>:6550
   --k3s-server-arg "--no-deploy=traefik"
   --k3s-server-arg "--tls-san=<VM_Public_IP>"
   --agents 2
   -v "/home/<your_user_name>/kube:/kube@agent[0,1]"
   -v "/home/<your_user_name>/kube:/kube@server[0]"
   --port "80:80@loadbalancer"

5. Define "local" clearly in below statement. Local laptop Vs AWS VM.

Verify the Cluster creation by running the following commands from your local machine where the kubectl is installed. It gives you the sample output as below if everything works fine.

6. kubeconfig file had a setting in the cluster property with 46687 as the port. Instructions say to only replace IP with public IP, nothing about port. Had to replace the port with 6550 after checking on the server directly. After that, ran the command to a access the cluster and getting an error

sudo kubectl get nodes --kubeconfig=myk3dconfig
Unable to connect to the server: x509: certificate signed by unknown authority. 

Solution: I had multiple kube contexts and kube config files since I attempted a local deploy and then followed it with a cloud deploy. Errors with kubernetes happening when there are multiple contexts enabled. Context switching does not happen even when the use-context param is passed. It ends up using some other context file. So use kubeconfig param for all kubectl commands and pass in path of relevant kubeconfig file. This can be mentioned as a troubleshooting/debugging step. Example below:

sudo kubectl delete pod kafka-0 -n kafka-cluster --kubeconfig=/home/ubuntu/kube/myk3dconfig
8. No golang installed on remote VM via AMI after Terraform config is done. Had to be manually installed. Solution: Include golang in the auto deploy scripts?
9. Port forwarding works just fine from local machine to remote k3d cluster.
10. Postman scripts from 3 down failed initially with a 400. Reason is that 127.0.0.1 is mapped to quickstart.local.digit but localhost somehow didn’t map onto quickstart.local.digit. Had already made change to the /etc/hosts file where 127.0.0.1 is mapped to quickstart.local.digit. Solution: Modified all postman APIs with localhost:8080 as the server URL. Users were created just fine.
11. As an additional convenience, scripts to delete above created users would be helpful. Then, these scripts can be run multiple times with setup and teardown phases.