feat: add common and console config values with comments (#1)

* feat: add common and console config values with comments

* refactor: abstract config for console

* chore: add .editorconfig

* feat: add rbac for migration

* refactor: move database url generation to global helpers

* feat: set default database name and database user name

* feat: add job to generate inter-service tokens

* feat: add bulker config values

* feat: abstract bulker config

* fix: fall back to global redis url

* feat: abstract rotor config

* fix: return backofflimit on migration to 1

* fix: uncomment default ingress rule so chart will deploy despite missing config

* feat: add ingest and syncctl config values

* refactor: strip prefixes from bulker config params

* refactor: strip prefixes from ingest and syncctl config options

* feat: add ingest config abstractions

* feat: add syncctl config abstractions

* refactor: simplify console env

* refactor: simplify bulker env

* fix: cast port to int

* feat: make deployments wait for token generation

* fix: missing values in auto-generate-tokens

* fix: missing ingest values in console env

* feat: add more wait init containers

* fix: don't use latest tag for images

* fix: cast port to int

* fix: set bulker config source token in console auth tokens

* fix: don't base64-encode the token hashes

* fix: update console values

* fix: disable redis auth

* refactor: simplify token generation

* refactor: use standard env format instead of custom environment

* feat: set internal nextauth url for environments where the public url can't be reached internally

* feat: add enable credentials login option

* docs: add some more comments

* refactor: add more wait init-containers to minimize unnecessary backoffs

* docs: update readme

* docs: small changes

* fix: redis service name in initcontainer

* refactor: don't capitalize acronyms in configuration names

* docs: expand readme

* docs: change syntax highlighting for env

* fix: secret key

* fix: set bulker url for console

* feat: bump appversion to 2.4.3

* fix: create base64 tokens for services other than the console

* feat: add role and role binding for syncctl

* refactor: rename auto-generate-tokens to token-generator and use own sa

* fix: update refrences to autogeneratetokens

* refactor: decouple primary service account from migration/token generation

* feat: add option to enable/disable creation of rbac for tokengenerator

* feat: enable running syncctl workloads in a different namespace

* fix: job names in wait-for-tokens init containers

* fix: allow token-generator unbridled secret creation

* fix: let syncctl check jobs

* feat: add proxy service for database in syncctl namespace

* fix: replace schema parameter in database url with search_path for syncctl

* refactor: move postgres auth out from global

* feat: enable auth for redis, provide default passwords for mongo and pg

* fix: redis url format

* fix: set global hash secret for ingest and bulker

* fix: add global hash secret env var prefix for ingest and bulker

* fix: set authsource in generated mongodb url

* feat: add proxy service for bulker as well

* fix: specify bulker_auth_token, not key for syncctl

* refactor: drop support for k8s: museum edition

* feat: use simplified, unified ingress by default

* docs: fix missing eof

* docs: add notice about disabling sign up

* feat: make syncctl url globally overridable

* docs: fix url parameters in readme

* fix: don't show password warning if the component is disabled

* docs: fix case

* feat: make config abstractions disableable

* feat allow setting env from secrets/configmaps

* feat: allow setting some configuration variables from configmaps/secrets

* feat: allow disabling config abstractions globally

* fix: forward batch ingestion to ingest (#3)

fix: forward batch ingest api calls to ingest

---------

Co-authored-by: Hendrik Heil <[email protected]>

.editorconfig

@@ -0,0 +1,3 @@
+[*]
+indent_style=space
+indent_size=2

Chart.yaml

@@ -4,7 +4,7 @@ type: application
 
 # Chart version is set automatically as part of the release process
 version: 0.0.0
-appVersion: 2.4.1
+appVersion: 2.4.3
 
 dependencies:
   - name: postgresql

README.md

@@ -1,8 +1,148 @@
 # Jitsu Helm Chart
-:warning: **This chart is under development and may receive breaking changes at any time.**
+
+## TL;DR
+```bash
+helm install jitsu oci://registry-1.docker.io/stafftasticcharts/jitsu -f-<<EOF
+ingress:
+  host: "jitsu.example.com"
+console:
+  config:
+    seedUserEmail: "[email protected]"
+    seedUserPassword: "changeMe"
+EOF
+```
+
+For a production deployment it is recommended to read through `values.yaml` and make conscious
+decisions in order to ensure the deployment is secure, reliable and scalable.
+
+## Basic Configuration
+`values.yaml`:
+```yaml
+postgresql:
+  auth:
+    password: "changeMe"
+mongodb:
+  auth:
+    passwords: ["changeMe"]
+redis:
+  auth:
+    password: "changeMe"
+
+ingress:
+  className: "nginx"
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt"
+  host: "jitsu.example.com"
+  tls: true
+
+console:
+  config:
+    # Populate with GitHub OAuth client credentials
+    githubClientId: "..."
+    githubClientSecret: "..."
+```
+
+Once you have logged in, set `console.config.disableSignup` to `true` to prevent anyone with a
+GitHub account from using your instance.
+
+See [values.yaml](values.yaml) for more configuration options.
 
 ## Dependencies
+This chart deploys the following dependencies by default in order to provide an easy out-of-the-box
+experience, however for production it is recommended you deploy these separately:
+
 * Postgres
 * Redis
 * Kafka
 * MongoDB
+
+In order to use your own instances of these, disable them in with their respective options:
+```yaml
+postgresql:
+  enabled: false
+redis:
+  enabled: false
+kafka:
+  enabled: false
+mongodb:
+  enabled: false
+```
+
+Then supply the connection details in the `config` section (or specifically per service):
+```yaml
+config:
+  databaseUrl: "postgres://..."
+  redisUrl: "redis://..."
+  kafkaBootstrapServers: "kafka:9092,..."
+  mongodbUrl: "mongodb://..."
+```
+
+## Configuration Options
+The individual services' configuration corresponds to the environment variables they accept. For
+services where every environment variable is prefixed with the service name, the prefix is stripped,
+otherwise the keys are naïvely converted to camel case, with each letter that would follow an
+underscore capitalized.
+
+Some values, in particular those that contain sensitive information or connection information, also
+allow you to reference a secret or configmap. In `values.yaml` these are suffixed with `From`. E.g.
+to read the database URL (`config.databaseUrl`) from a secret, set it as you would an environment
+variable:
+
+```yaml
+config:
+  databaseUrlFrom:
+    secretKeyRef:
+      name: database-secret-name
+      key: database-url-key
+```
+
+For the full list of variables that support this syntax, see `values.yaml`.
+
+Many of the configuration values will be set automatically when left empty, such as connection
+parameters for services deployed by the subcharts, tokens and URLs for inter-service communication
+and values that can be directly derived from other values. When this is the case it is noted in the
+comments above the value. Links are also provided to relevant upstream documentation.
+
+Some configuration values contain structured data. For these you can either specify them as a string
+as you would in an environment variable, or as a dict that will be converted to the appropriate
+string representation by the chart.
+
+One notable example of this, and the only exception to the 1:1 mapping of environment variables to
+camel cased keys, is the `bulker.config.destination` value. The Bulker takes an arbitrary number of
+destination environment variables in the form of `BULKER_DESTINATION_*`. These are represented in
+`values.yaml` as a dict of either strings or dicts.
+
+Example:
+
+```yaml
+bulker:
+  config:
+    destination:
+      postgres:
+        id: postgres
+      s3: '{"id":"s3"}'
+```
+
+Becomes:
+
+```bash
+BULKER_DESTINATION_POSTGRES='{"id":"postgres"}'
+BULKER_DESTINATION_S3='{"id":"s3"}'
+```
+
+If you prefer to configure one or more services manually through the environment, you can disable
+the configuration abstractions by setting `config.enabled` to `false`, either at the top-level or
+service-level.
+
+## Inter-Service Authentication
+The different Jitsu services communicate with each other using tokens and corresponding salted
+hashes to verify. These can be managed manually, however by default they are generated by a job and
+stored in a secret. Each service then gets access to the tokens they need through the environment.
+
+In order to disable this, set `tokenGenerator.enabled` to `false` and supply the tokens manually.
+
+## Running Connectors in a Different Namespace
+By default syncctl runs connectors in the same namespace as the rest of the Jitsu services. If you
+wish to run these ephemeral and to some degree user-controlled workloads in a separate namespace you
+can set `syncctl.config.kubernetesNamespace` to the desired namespace, and the chart will create the
+namespace, service proxies for the bulker and databse, and the necessary RBAC resources for you.

templates/NOTES.txt

@@ -0,0 +1,23 @@
+{{- $warned := false }}
+{{- define "jitsu.warn-default-password" }}
+{{- printf "\033[1;31m" -}}
+WARNING: You are using the default password for {{ . }}.
+{{ printf "\033[0m" -}}
+{{- end }}
+{{- if and .Values.redis.enabled (eq .Values.redis.auth.password "jitsu") }}
+{{- $warned = true }}
+{{- include "jitsu.warn-default-password" "Redis" }}
+{{- end }}
+{{- if and .Values.postgresql.enabled (eq .Values.postgresql.auth.password "jitsu") }}
+{{- $warned = true }}
+{{- include "jitsu.warn-default-password" "PostgreSQL" }}
+{{- end }}
+{{- if and .Values.mongodb.enabled (eq (index .Values.mongodb.auth.passwords 0) "jitsu") }}
+{{- $warned = true }}
+{{- include "jitsu.warn-default-password" "MongoDB" }}
+{{- end }}
+{{- if $warned }}
+{{- printf "\033[1;32m" -}}
+For production use, change the passwords in values.yaml.
+{{- printf "\033[0m" -}}
+{{- end }}

templates/_helpers.tpl

@@ -60,3 +60,60 @@ Create the name of the service account to use
 {{- default "default" .Values.serviceAccount.name }}
 {{- end }}
 {{- end }}
+
+{{- define "jitsu.publicUrl" }}
+{{- if .Values.ingress.enabled }}
+{{- printf "http%s://%s%s"
+  (.Values.ingress.tls | ternary "s" "")
+  .Values.ingress.host
+  (not .Values.ingress.port | ternary "" (printf ":%s" .Values.ingress.port))
+-}}
+{{- end }}
+{{- end }}
+
+{{- define "jitsu.databaseUrl" -}}
+{{- if and (not .Values.config.databaseUrl) .Values.postgresql.enabled }}
+{{- with $.Values.postgresql.auth -}}
+{{ printf "postgres://%s:%s@%s:%d/%s?schema=newjitsu"
+  .username
+  .password
+  (printf "%s-postgresql" $.Release.Name)
+  5432
+  .database
+}}
+{{- end }}
+{{- else -}}
+{{ .Values.config.databaseUrl }}
+{{- end }}
+{{- end }}
+
+{{- define "jitsu.redisUrl" -}}
+{{- if and (not .Values.config.redisUrl) .Values.redis.enabled }}
+{{- with $.Values.redis -}}
+{{ printf "redis://:%s@%s:%d"
+  .auth.password
+  (printf "%s-redis-master" $.Release.Name)
+  6379
+}}
+{{- end }}
+{{- else -}}
+{{ .Values.config.redisUrl }}
+{{- end }}
+{{- end }}
+
+{{- define "jitsu.mongodbUrl" -}}
+{{- if and (not .Values.config.mongodbUrl) .Values.mongodb.enabled }}
+{{- with $.Values.mongodb.auth -}}
+{{ printf "mongodb://%s:%s@%s:%d/%s?authSource=%s"
+  (index .usernames 0)
+  (index .passwords 0)
+  (printf "%s-mongodb" $.Release.Name)
+  27017
+  (index .databases 0)
+  (index .databases 0)
+}}
+{{- end }}
+{{- else -}}
+{{ .Values.config.mongodbUrl }}
+{{- end }}
+{{- end }}