Merge branch 'main' into onlyoffice-documentserver-ee

README.md

@@ -1,131 +1,126 @@
-# fairagro-playground #
+# FAIRagro basic_infrastructure #
 
 ## How to deal with this project ##
 
-This is a kubernetes project. You will need to have the following
-tools available to deal with it:
+This project delivers a `vscode` dev container that contains all tools needed. To use that container
+you need a running docker installation and choose 'Clone in Volume' after opening it in `vscode`.
+It will take quite some time to build the container.
 
-- [kubectl](https://kubernetes.io/de/docs/tasks/tools/install-kubectl/)
-- [helm](https://helm.sh/)
-- [helmfile](https://github.com/helmfile/helmfile)
-- [sops](https://github.com/getsops/sops#sops-secrets-operations)
-- [gpg](https://gnupg.org/)
-- [openssl](https://www.openssl.org/)
-- [helm-secrets](https://github.com/jkroepke/helm-secrets)
+As the project contains encrypted secrets (that are also instantiated when running the container),
+you will need to enter your personal `gpg` passphrase. This of course requires that your pgp key
+is registered within the project. If not, please ask a colleague to add your key. Any colleague
+already registered for the project will be able to do so. You can find the list of registered people
+within the `public_gpg_keys` folder of an `environment` (so every environment has his own list of
+registered users).
 
-Currently the project is being managed from Windows Powershell. So
-this is the language of the scripts you find in the `scripts`
-directory. All tools are also available for other platforms so you
-are not forced to use Windows, though.
-
-### `kubectl` ###
-
-This is teh stabdard kubernetes command line tool for.
-
-### `helm` ###
-
-This is the kubernetes package manager, `helm` packages are called
-`charts`. We use `helm` for third party packages and we also write
-our own `helm` `charts` which can be found in the directory
-`helmcharts`.
-
-### `helmfile` ###
-
-This describes the whole project within a single file -- the
-`helmfile`. So it can be considers as project management tool. The
-`helmfile` references all `helm` `charts` as well as the
-correspoding configuration and secrets.
-
-### `sops` ###
-
-This tool is used for secret encrytion. It interfaces lower level
-encryption tools like `gpg` and allows for multiuser features and
-better `git` integration.
-
-### `gpg` ###
-
-This is the low-level encryption tool used by sops when involving
-developer keys. Each developer has his or her own `gpg` key pair.
-`sops` will encrypt all secrets using its a file-dependent master
-key which in turn will be encrypted by the `gpg` keys of all
-involved developers.
-
-### `openssl` ###
-
-`openssl` is needed when creating new kubernetes user accounts --
-as in fact a kubernetes user is nothing more than as an X509
-certificate signed by the kubernetes server.
-
-### `helm-secrets` ###
-
-This tool allows for the integration of `helm` and `sops`.
-
-## Contexts, Environments and Users ##
-
-This project is developed with kubernetes contexts in mind. A
-context consists
-of a kubernetes server, a namespace (usually the namespace
-will be `fairagro`) and a user account. All components of this
-project are always installed into the same namespace.
-You can consider the combination of server and namespace as an
-environment, each with its own
-configurations and secrets. Currently the environment
-'corki/fairagro'
-is used for development/staging while the environment
-'draven/fairagro' is used for production.
-
-All active environments are kept with in the project in the
-`environments` directory.
-
-Each user (aka developer) will need to manage his own kubeconfig
-files that include his personal access keys for kubernetes. The
-included Windows Powershell scripts expect the kubeconfig file to
-be stored in the folder
-`%USERPROFILE%\.k8s_config\<servername>_<namespace>`.
-So there will be one kubeconfig file per environment. It is
-expected that each developer will only have one user account per
-environment. This user will have admin right in the corresponding
-namespace.
-
-You can change the current context with the script
-`set_context.ps1`. Note that it needs to be dot-sourced. E.g.:
-
-    . scripts/set_context.ps1 corki fairagro
-
-This script will also load all public gpg keys used by `sops` into
-your `gpg` keyring and set command aliases for `kubectl`, `helm`,
-`helmfile` and `sops`.
-
-## Secrets management ##
-
-Secrets are managed per environment (as other configuration). You
-will
-find a `secrets` folder within each environment. Secrets files are
-encrypted with `sops` and always end in '.enc.yaml'. They are
-referenced in the 'helmfile.yaml' for each release.
-
-`sops` manages secret encryption using `gpg` for each developer.
-There will also be secret encryption for the continuous deployment
-pipeline which will probably use `age`.
-
-Within each environment folder you will find a file '.sops.yaml'
-that defines the behavior of `sops` for all files within the
-directory as well as all sub-directories. You will notice that
-files ending in '.enc.yaml' are encrypted using the gpg keys of all
-active developers. Note that only the public keys of the developers
-are needed for encryption, so every developer can encrypt secrets
-for all other developers (as well as the CD pipeline). Only for
-decryption the private key will be required. Note also that the
-public keys of all active developers are included in the
-`public_keys` folder of each environment. These keys will be loaded
-automatically into your `gpg` keyring when you use the script
-`set_context.ps1`.
-
-Currently secrets files created by extracting secrets from
-a 'values.yaml' file as needed by `helm`. These secrets are copied
-to a corresponding secret file that has to be encrypted using
-`sops`. Secret files are merged into the helm charts values upon
-execution of `helm`/`helmfile`. If you use VisualStudio Code you
-can install the 'vscode-sops' plugin that will automatically
-encrypt all files according to the '.sops.yaml' file.
+## Add a new user to the project ##
 
+To add a new user to the project, please peform these steps.
+
+On the machine where you have generated your key:
+
+* Find out the key fingerprint:
+
+  ```bash
+  userid=Any string that identifies yor gpg key
+  fingerprint=$(gpg --list-keys -a $userid | sed '2!d' | tr -d " ")
+  echo $fingerprint
+  ```
+
+  `userid` can be any sub-string that identifies your key, i.e. that is listed behind 'uid' when
+  executing `gpg --list-keys` without the `-a` switch. In this example it coud be 'carsten':
+
+  ```bash #2
+  pub   ed25519 2023-07-24 [SC]
+      CC7B10CE8D78010ABB043F8DB1C462E90012ECFE # the fingerprint
+  uid           [ultimate] Carsten Scharfenberg <[email protected]>  # the full user id
+  sub   cv25519 2023-07-24 [E]
+  ```
+
+* Export the public key:
+
+  ```bash
+  username=Your_Name
+  filename=${fingerprint}_${username}.asc
+  gpg --export --armor -o $filename $fingerprint
+  ```
+
+  In this case `username` is the actual user name consisting of capitalized first and last name,
+  divided by an underscore. E.g.: `Carsten_Scharfenberg`.
+
+With in this project structure on any machine/dev container:
+
+* Import the key manually by:
+
+  ```bash
+  gpg --import '<filename>`
+  ```
+
+* cd into an environment folder
+* Add the public `gpg` key file to the folder `public_gpg_keys` without changing the filename
+* Add the key fingerprint to the file `.sops.yaml`
+* Re-encrypt all secret files:
+
+  ```bash
+  find . -name *.enc.yaml | xargs sops updatekeys
+  ```
+
+## `kubectl` cheat sheet ##
+
+find some useful `kubectl` calls below.
+
+### List everything within a namespace ###
+
+The normal call to list 'everything' within a namespace is, e.g.:
+
+```bash
+kubectl get all -n fairagro-nextcloud
+```
+
+This won't really list everything. Instead you can use:
+
+```bash
+kubectl api-resources --namespaced=true --verbs=list -o name \
+| xargs -n 1 kubectl get --show-kind --ignore-not-found -n fairagro-nextcloud
+```
+
+### Delete 'everything' from a namespace ###
+
+To delete everything from a namespace you would normally call, e.g.
+
+```bash
+kubectl delete all --all -n fairagro-nextcloud
+```
+
+In fact this does not delete everything, but keeps, e.g., `ConfigMap`s, `Secret`s and
+`PersistentVolumeClaim`s. You also could delete the whole namespace to really delete everything:
+
+```bash
+kubectl delete namespace fairagro-nextcloud
+```
+
+Only this is not ideal for this project, because it will also delete `Roles`s and `RoleBinding`s
+that are created by the `zalf-rdm/misc` project and are thus not easy to re-create.
+
+Instead this is a good alternative:
+
+```bash
+kubectl delete all,pdb,configmap,secret,pvc,ingress,serviceaccount --all -n fairagro-nextcloud
+```
+
+Consider to [list all resources](#list-everything-within-a-namespace) to check if really
+everything except `Roles`s and `RoleBindings` has been deleted. Note that kubernetes will
+automatically recreate `configmap/kube-root-ca.crt` and `serviceaccount/default`.
+
+### Access kuberenetes cluster using `etcdctl` ###
+
+I case you would like to access the underlying etcd database of kuberentes directly, log
+into one of the kubernetes nodes. Then:
+
+```bash
+sudo -i
+set -a
+. /etc/ssl/etdctl
+set +a
+/usr/local/bin/etcdctl get / --prefix --keys-only  # an example
+```

ansible/requirements.txt

@@ -1,4 +1,4 @@
-ansible==10.3.0
+ansible==10.4.0
 ansible-core==2.17.6
 cffi==1.17.0
 cryptography==43.0.1

ansible/roles/reverse_proxy/templates/nginx.conf.j2

@@ -19,9 +19,9 @@ events {
 stream {
 
     map $ssl_preread_server_name $name {
-{%- for target in reverse_proxy__forwarding_targets -%}
+{% for target in reverse_proxy__forwarding_targets %}
         {{ target.fqdn }} {{ target.ip | default(reverse_proxy__default_ip) }}:{{ target.port | default(reverse_proxy__default_port) | string }};
-{%- endfor -%}
+{% endfor %}
     }
 
     server {

environments/draven/values/zalf-nextcloud.yaml

@@ -1,6 +1,9 @@
+# allow internal IPs for reverse proxy and collabora.
+# All traffic stems from ingress-nginx, which is internal to the cluster.
 allowed_collabora_ips:
-  - 161.97.79.221 # collabora.fairagro.net on external reverse proxy
-allowed_reverse_proxy_ips: 161.97.79.221 # the extrenal reverse proxy IP
+  - 172.16.64.0/18
+allowed_reverse_proxy_ips: 172.16.64.0/18
+
 postgres_db:
   size: 4Gi
   storageClass: postgres