Fully separated App-of-Apps now for crossplane with and without Exter…

…nal Secrets Operator for later clarity in blogs etc.

.github/workflows/crossplane-argocd-external-secrets.yml

@@ -39,7 +39,7 @@ jobs:
       - name: Use ArgoCD's AppOfApps pattern to deploy all Crossplane components
         run: |
           echo "--- Let Argo do it's magic installing all Crossplane components"
-          kubectl apply -n argocd -f argocd/crossplane-app-of-apps.yaml 
+          kubectl apply -n argocd -f argocd/crossplane-eso-app-of-apps.yaml 
 
       - name: Check crossplane status
         run: |

.github/workflows/crossplane-argocd.yml

@@ -57,7 +57,7 @@ jobs:
       - name: Check crossplane status
         run: |
           echo "--- Wait for crossplane to become ready (now prefaced with until as described in https://stackoverflow.com/questions/68226288/kubectl-wait-not-working-for-creation-of-resources)"
-          until kubectl wait --for=condition=PodScheduled pod -l app=crossplane --namespace crossplane-system --timeout=120s; do : ; done
+          until kubectl wait --for=condition=PodScheduled pod -l app=crossplane --namespace crossplane-system --timeout=120s > /dev/null 2>&1; do : ; done
           kubectl wait --for=condition=ready pod -l app=crossplane --namespace crossplane-system --timeout=120s
 
           echo "--- Wait until AWS Provider is up and running (now prefaced with until to prevent Error from server (NotFound): providers.pkg.crossplane.io 'provider-aws-s3' not found)"

README.md

@@ -794,6 +794,28 @@ And as I like to show solutions that are fully cromprehensible - ideally without
 
 So I thought the exact secret management tool I use in this case is not that important and I trust my readers that they will choose the provider that suites them the most. That beeing said I chose [Doppler](https://www.doppler.com/) with their [generous free Developer plan](https://www.doppler.com/pricing).
 
+As External-Secrets introduce more complexity to our setup, I decided to divide the crossplane only solution from the more advanced using External Secrets Operator. Therefore the `argocd` directory now looks like this:
+
+```shell
+$ tree                                                                                                                                                  ✔ 
+.
+├── applications
+│   ├── crossplane-core.yaml
+│   ├── crossplane-helm-secret.yaml
+│   └── crossplane-provider-aws.yaml
+├── applications-eso
+│   ├── crossplane-core.yaml
+│   ├── crossplane-helm-secret.yaml
+│   ├── crossplane-provider-aws.yaml
+│   ├── external-secrets-config.yaml
+│   └── external-secrets-operator.yaml
+├── crossplane-app-of-apps.yaml
+├── crossplane-eso-app-of-apps.yaml
+...
+```
+
+Where `applications` and the corresponding `crossplane-app-of-apps.yaml` feature the crossplane only solution - and `applications-eso` with it's `crossplane-eso-app-of-apps.yaml` App-of-Apps counterpart feature the more advanced ESO solution.
+
 
 
 ### Create multiline Secret in Doppler
@@ -861,7 +883,7 @@ dependencies:
     version: 0.9.11
 ```
 
-Now telling ArgoCD where to find our simple external-secrets Helm Chart, we again use Argo's `Application` manifest in [argocd/applications/external-secrets-operator.yaml](argocd/applications/external-secrets-operator.yaml):
+Now telling ArgoCD where to find our simple external-secrets Helm Chart, we again use Argo's `Application` manifest in [argocd/applications-eso/external-secrets-operator.yaml](argocd/applications-eso/external-secrets-operator.yaml):
 
 ```yaml
 # The ArgoCD Application for external-secrets-operator
@@ -990,7 +1012,7 @@ spec:
 Although we created a `CREDS` secret in Doppler, we need to use `path: creds` here - since we use the ClusterSecretStore name transformer `lower-snake`! Otherwise we get reconcile errors, since the `ExternalSecret` looks for the uppercase path!
 
 
-We also need to create a ArgoCD Application so that Argo will deploy both `ClusterSecretStore` and `ExternalSecret` for us :) Therefore I created [`argocd/applications/external-secrets-config.yaml`](argocd/applications/external-secrets-config.yaml):
+We also need to create a ArgoCD Application so that Argo will deploy both `ClusterSecretStore` and `ExternalSecret` for us :) Therefore I created [`argocd/applications-eso/external-secrets-config.yaml`](argocd/applications-eso/external-secrets-config.yaml):
 
 ```yaml
 # The ArgoCD Application for external-secrets-operator
@@ -1055,7 +1077,7 @@ Therefore let's give our `external-secrets-config` more `syncPolicy.retry.limit`
 
 ### Point the Crossplane AWS ProviderConfig to our External Secret created Secret from Doppler
 
-Therefore we need to change our [`upbound/provider-aws-s3/provider/provider-config-aws.yaml`](upbound/provider-aws-s3/provider/provider-config-aws.yaml) to use another Secret name and namespace:
+We need to change our `ProviderConfig` at [`upbound/provider-aws/provider-eos/provider-config-aws.yaml`](upbound/provider-aws/provider-eos/provider-config-aws.yaml) to use another Secret name and namespace: 
 
 ```yaml
 apiVersion: aws.upbound.io/v1beta1
@@ -1067,7 +1089,7 @@ spec:
     source: Secret
     secretRef:
       namespace: external-secrets
-      name: aws-creds
+      name: aws-secrets-from-doppler
       key: creds
 ```
 
@@ -1139,7 +1161,7 @@ jobs:
       - name: Use ArgoCD's AppOfApps pattern to deploy all Crossplane components
         run: |
           echo "--- Let Argo do it's magic installing all Crossplane components"
-          kubectl apply -n argocd -f argocd/crossplane-app-of-apps.yaml 
+          kubectl apply -n argocd -f argocd/crossplane-eso-app-of-apps.yaml 
 
       - name: Check crossplane status
         run: |

argocd/applications-eso/crossplane-core.yaml

@@ -0,0 +1,31 @@
+# The ArgoCD Application for crossplane core components themselves
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: crossplane-core
+  namespace: argocd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+  annotations:
+    argocd.argoproj.io/sync-wave: "3"
+spec:
+  project: default
+  source:
+    repoURL: https://github.com/jonashackt/crossplane-argocd
+    targetRevision: HEAD
+    path: crossplane
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: crossplane-system
+  syncPolicy:
+    automated:
+      prune: true    
+    syncOptions:
+    - CreateNamespace=true
+    retry:
+      limit: 1
+      backoff:
+        duration: 5s 
+        factor: 2 
+        maxDuration: 1m

argocd/applications-eso/crossplane-helm-secret.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: crossplane-helm-repo
+  namespace: argocd
+  labels:
+    argocd.argoproj.io/secret-type: repository
+  annotations:
+    argocd.argoproj.io/sync-wave: "2"
+stringData:
+  name: crossplane
+  url: https://charts.crossplane.io/stable
+  type: helm 

argocd/applications-eso/crossplane-provider-aws.yaml

@@ -0,0 +1,33 @@
+# The ArgoCD Application for all Crossplane AWS providers incl. it's ProviderConfig
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: provider-aws
+  namespace: argocd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+  annotations:
+    argocd.argoproj.io/sync-wave: "4"
+spec:
+  project: default
+  source:
+    repoURL: https://github.com/jonashackt/crossplane-argocd
+    targetRevision: HEAD
+    path: upbound/provider-aws/provider-eos
+  destination:
+    namespace: default
+    server: https://kubernetes.default.svc
+  # Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with
+  # 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'
+  syncPolicy:
+    automated:
+      prune: true    
+    retry:
+      # Using limit 5, so that the ProviderConfig can "wait" (via retry) for the Provider and it's CRDs to be deployed
+      # and not to run into 'The Kubernetes API could not find aws.upbound.io/ProviderConfig for requested resource default/default.'
+      limit: 5
+      backoff:
+        duration: 5s 
+        factor: 2 
+        maxDuration: 1m

argocd/applications/crossplane-core.yaml

@@ -8,7 +8,7 @@ metadata:
   finalizers:
     - resources-finalizer.argocd.argoproj.io
   annotations:
-    argocd.argoproj.io/sync-wave: "3"
+    argocd.argoproj.io/sync-wave: "1"
 spec:
   project: default
   source:

argocd/applications/crossplane-helm-secret.yaml

@@ -6,7 +6,7 @@ metadata:
   labels:
     argocd.argoproj.io/secret-type: repository
   annotations:
-    argocd.argoproj.io/sync-wave: "2"
+    argocd.argoproj.io/sync-wave: "0"
 stringData:
   name: crossplane
   url: https://charts.crossplane.io/stable

argocd/applications/crossplane-provider-aws.yaml

@@ -8,7 +8,7 @@ metadata:
   finalizers:
     - resources-finalizer.argocd.argoproj.io
   annotations:
-    argocd.argoproj.io/sync-wave: "4"
+    argocd.argoproj.io/sync-wave: "2"
 spec:
   project: default
   source:

argocd/crossplane-eso-app-of-apps.yaml

@@ -0,0 +1,29 @@
+# The ArgoCD App of Apps for all Crossplane components
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: crossplane
+  namespace: argocd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: default
+  source:
+    repoURL: https://github.com/jonashackt/crossplane-argocd
+    targetRevision: HEAD
+    path: argocd/applications-eso
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: crossplane-system
+  syncPolicy:
+    automated:
+      prune: true    
+    syncOptions:
+    - CreateNamespace=true
+    retry:
+      limit: 1
+      backoff:
+        duration: 5s 
+        factor: 2 
+        maxDuration: 1m

upbound/provider-aws/provider-eos/provider-aws-s3.yaml

@@ -0,0 +1,9 @@
+apiVersion: pkg.crossplane.io/v1
+kind: Provider
+metadata:
+  name: provider-aws-s3
+spec:
+  package: xpkg.upbound.io/upbound/provider-aws-s3:v0.46.0
+  packagePullPolicy: Always
+  revisionActivationPolicy: Automatic
+  revisionHistoryLimit: 1

upbound/provider-aws/provider-eos/provider-config-aws.yaml

@@ -0,0 +1,11 @@
+apiVersion: aws.upbound.io/v1beta1
+kind: ProviderConfig
+metadata:
+  name: default
+spec:
+  credentials:
+    source: Secret
+    secretRef:
+      namespace: external-secrets
+      name: aws-secrets-from-doppler
+      key: creds

upbound/provider-aws/provider/provider-config-aws.yaml

@@ -6,6 +6,6 @@ spec:
   credentials:
     source: Secret
     secretRef:
-      namespace: external-secrets
+      namespace: crossplane-system
       name: aws-creds
       key: creds