Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CH-100 Add preliminary docker-compose support #735

Merged
merged 102 commits into from
Sep 17, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
d25230f
CH-100 Add first code to call a dedicated docker-compose generation
aranega Feb 6, 2024
6f306a1
Squashed commit of the following:
aranega Feb 7, 2024
6bbae19
CH-100 Add first skaffold dedicated generation for docker compose
aranega Feb 7, 2024
5287545
CH-100 Make skaffold script a little bit more generic (ugly)
aranega Feb 7, 2024
9f75c9c
CH-100 Fix issue with entrypoint
aranega Feb 7, 2024
c159a4f
CH-100 Remove generation of chart files for docker-compose
aranega Feb 8, 2024
02bd318
CH-100 Rollback default per-app helm template directory
aranega Feb 9, 2024
1a57064
CH-100 Add subdomain configuration
aranega Feb 9, 2024
b97c19c
CH-100 Fix bad "links" generation
aranega Feb 9, 2024
85dcfd9
CH-100 Add support for aliases and service links
aranega Feb 12, 2024
1ac8226
CH-100 Add first support for auto databases
aranega Feb 14, 2024
56e22b7
CH-100 Add finer grain port handling
aranega Feb 14, 2024
336b558
CH-100 Change way port is exposed to outside world
aranega Feb 15, 2024
a004ffb
CH-100 Fix issue with env var quoting
aranega Feb 15, 2024
f73108c
CH-100 Add special behavior to produce allvalues.yaml
aranega Feb 15, 2024
2370b20
CH-100 Add actual docker-compose.yaml generation in harness-deployment
aranega Feb 15, 2024
5ed630d
CH-100 Add first dedicated templates for postgres
aranega Feb 15, 2024
111a4f3
CH-100 Add volumes for db
aranega Feb 15, 2024
ce4596a
CH-100 Add first template for events
aranega Feb 16, 2024
3868b56
CH-100 Refactor templates for the db
aranega Feb 16, 2024
a0476d6
CH-100 Add /etc/hosts generation
aranega Feb 16, 2024
bd3e423
CH-100 Add specific varenvs by container
aranega Feb 16, 2024
8851c48
CH-100 Rename some templates
aranega Feb 20, 2024
ced7cd2
CH-100 Add first traeffik configuration
aranega Feb 20, 2024
f7ebeb0
CH-100 Fix bad indentation in docker-compose.yaml
aranega Feb 20, 2024
8fdcba6
CH-100 Remove exposition of ports
aranega Feb 20, 2024
428a619
CH-100 Add post-process mechanism to generate files
aranega Feb 21, 2024
9d6c8b2
CH-100 Add new templates
aranega Feb 21, 2024
ae7cf19
CH-100 Add new templates (not modified yet)
aranega Feb 21, 2024
7acc7f5
CH-100 Add generation of resources files
aranega Feb 22, 2024
571c2ab
CH-100 Add dependency between service and db
aranega Feb 22, 2024
9c8c19e
CH-100 Add secret handling
aranega Feb 23, 2024
f6d0c67
CH-100 Remove argo from dependencies
aranega Feb 23, 2024
4fd6dee
CH-100 Change from pyyaml to ruamel
aranega Feb 23, 2024
86c0e4f
CH-100 Remove tmp "events" from deps
aranega Feb 23, 2024
a5909ce
CH-100 Add back sc.yaml
aranega Feb 23, 2024
4379252
CH-100 Add first handling of NFS volume
aranega Feb 23, 2024
46ee140
CH-100 Add volumes conditionnally
aranega Feb 26, 2024
e135c6a
CH-100 Add first tutorial about docker compose
aranega Feb 26, 2024
0f70146
CH-100 Add first real config for gatekeeper
aranega Feb 29, 2024
6bbef64
CH-100 Add reverse proxy config for gatekeeper
aranega Feb 29, 2024
3b437cf
CH-100 Fix path to generated resources
aranega Feb 29, 2024
1135b99
CH-100 Fix bad GK configuration
aranega Mar 1, 2024
b23f03b
CH-100 Adapt gk configuration
aranega Mar 1, 2024
361ca46
CH-100 Fix gk configuration
aranega Mar 1, 2024
74d074e
CH-100 Comment unused "ports"
aranega Mar 1, 2024
fec25be
CH-100 Fix issue with service names in dependencies
aranega Mar 15, 2024
0db12d8
CH-100 Fix issue with db-volumes
aranega Mar 15, 2024
f8d4c84
CH-100 Fix health check for account services
aranega Mar 15, 2024
4121180
CH-100 Fix issue with dependencies
aranega Mar 15, 2024
d81abc2
Merge branch 'develop' of github.com:MetaCell/cloud-harness into feat…
aranega Mar 20, 2024
5bf5532
CH-100 Change tagPolicy for docker-compose target
aranega Mar 26, 2024
319352b
CH-100 Add first information for the docker compose target
aranega Mar 26, 2024
7ec48e1
CH-100 Update documentation
aranega Mar 27, 2024
d85389a
Merge branch 'develop' into feature/CH-100-merge
aranega Mar 27, 2024
7edfcc4
#CH-100 improve documentation
filippomc Mar 27, 2024
e5d56b7
CH-100 First refactoring
aranega Apr 1, 2024
368725a
CH-100 Clean imports/comments
aranega Apr 1, 2024
a4cd813
CH-100 Refactor helm test
aranega Apr 1, 2024
1d63bc4
CH-100 Add tests for docker compose target
aranega Apr 1, 2024
b125dfc
CH-100 Add tests for docker compose
aranega Apr 1, 2024
891b237
Merge branch 'feature/CH-100' of github.com:MetaCell/cloud-harness in…
aranega Apr 1, 2024
2c7e264
CH-100 Fix type hinting for Python 3.9
aranega Apr 1, 2024
866dee9
CH-100 Add conditional test for docker compose if "helm" is installed
aranega Apr 1, 2024
f226829
CH-100 Update scripts
aranega Apr 2, 2024
ac4b863
CH-100 Fix imports of harness-deployment
aranega Apr 2, 2024
9b4dcaa
CH-100 Fix issue with TAG policy in skaffold for docker compose
aranega Apr 18, 2024
10f2c0f
CH-100 Update configuration for db in docker compose
aranega Apr 19, 2024
83d9b8a
CH-100 Add support to link databases with service in docker compose
aranega Apr 19, 2024
e5361ca
CH-100 Change location of traefik config files
aranega Apr 19, 2024
4c5d369
CH-100 Add specific option to expose locally a DB
aranega Apr 20, 2024
6cf0a83
Merge branch 'develop' of github.com:MetaCell/cloud-harness into feat…
filippomc Jul 30, 2024
fba11c6
Merge branch 'develop' of github.com:MetaCell/cloud-harness into feat…
filippomc Sep 12, 2024
4f5f34c
CH-110 fix dc resources mapping when not numbers
filippomc Sep 12, 2024
bbc57d2
chore: remove unneded dependencies
filippomc Sep 12, 2024
2a87ba5
CH-110 small fixes + tests
filippomc Sep 13, 2024
5adab28
CH-110 chore: ignore generated compose file
filippomc Sep 13, 2024
c10719e
CH-126 update python client generation
filippomc Sep 13, 2024
b344129
CH-110 improve output on error
filippomc Sep 13, 2024
1ddd023
CH-126 improve code generation
filippomc Sep 13, 2024
97ff173
CH-110 improve output on error
filippomc Sep 13, 2024
a5477a9
CH-110 fix tests for docker-compose
filippomc Sep 13, 2024
e5fb656
Merge branch 'develop' of github.com:MetaCell/cloud-harness into feat…
filippomc Sep 13, 2024
501ad71
Merge branch 'feature/CH-147-lint' of github.com:MetaCell/cloud-harne…
filippomc Sep 13, 2024
f69b4dd
Merge branch 'develop' of github.com:MetaCell/cloud-harness into feat…
filippomc Sep 13, 2024
8bdb619
CH-66 change secrets in configmap
filippomc Sep 16, 2024
ea20917
Fix merge issues
filippomc Sep 16, 2024
4fe5c4c
Remove wrong dependency
filippomc Sep 16, 2024
b9c3979
CH-147 fix linting issue
filippomc Sep 16, 2024
e838008
CH-17 fix some dependencies
filippomc Sep 16, 2024
4a04180
CH-17 fix python backend debug
filippomc Sep 16, 2024
5a4700f
Update test CI/CD
filippomc Sep 16, 2024
65bcc2b
CH-17 fix build issue
filippomc Sep 16, 2024
1562e95
CH-17 fix broken API tests
filippomc Sep 16, 2024
0f88e07
Improve testing docs
filippomc Sep 16, 2024
2535a16
Improve cookie auth automations
filippomc Sep 16, 2024
18c47cd
Lint fixes
filippomc Sep 17, 2024
0eae79b
Lint fixes
filippomc Sep 17, 2024
3529200
CH-110 small changes to docker compose generation
filippomc Sep 17, 2024
08b9645
CH-100 refactoring, small fixes
filippomc Sep 17, 2024
7eb8ca3
Linting fix
filippomc Sep 17, 2024
8bb9a1b
Linting fix
filippomc Sep 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
CH-100 Add first code to call a dedicated docker-compose generation
  • Loading branch information
aranega committed Feb 6, 2024
commit d25230ff5144cfead4bf98eb343ddd684dfa654a
22 changes: 22 additions & 0 deletions deployment-configuration/compose/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
10 changes: 10 additions & 0 deletions deployment-configuration/compose/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
apiVersion: v1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same file used on Kubernetes, do we need it to be duplicated?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your point, but trying to share the files between the helm configuration files and the docker compose one will only make things more obscure to understand.

Those two files (this one and the values.yaml) that are in the deployment-configuration needs to be duplicated in a way. If not, it means that there is a weird behavior that we should implement saying that the content of those files that are required for the template generation to work, need to be took from the helm folder for some reasons. It also means that we cannot rely on the current mechanism which copies files for a target from the corresponding folder, but instead, we would have a mix between "I pick files from the dedicated compose folder, but sometimes no, I need to pick them in the helm folder", which makes the generator more complicated and the logic of the configuration file generation more obscure.

A lot of files appears to be duplicated, but they are here to support the simplicity of "when I generate a target, the files from the corresponding target folder are copied" instead of having a mix of "sometimes its took from this folder, sometimes no", which would also require to implement a mechanism to check if the files that needs to be picked from the helm folder needs to be copied only if it doesn't exists in the compose folder to support configuration overriding (or other target folder if we support others in the future).

Another solution would be to share those files in a common or shared-configuration folder which would require to implement the same logic described above, but with another folder. I would vote against it as it makes the whole logic more complicated to understand, makes the generator more complex than just "copy files from the corresponding folder" and breaks, in a way, the isolation and independence of each target, to avoid having two files that looks the same. Why make it complicated when we can keep it simple and stupid?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your point the reason to avoid duplication. I'm removing the irrelevant configuration from compose's values.yaml

appVersion: "0.0.1"
description: CloudHarness Helm Chart
name: cloudharness
version: 0.0.1
maintainers:
- name: Filippo Ledda
email: filippo@metacell.us
- name: Zoran Sinnema
email: zoran@metacell.us
4 changes: 4 additions & 0 deletions deployment-configuration/compose/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# CloudHarness Helm chart: deploy CloudHarness to k8s

Helm is used to define the CloudHarness deployment on Kubernetes. For further information about Helm, see https://helm.sh.

103 changes: 103 additions & 0 deletions deployment-configuration/compose/templates/auto-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
version: '3.7'

services:
{{- range $app_name, $app_config := .Values.apps }}
{{- if has $app_name (list "argo" "nfsserver" "workflows" "events" ) }} {{- /* We deactivate generation for some services */}}
{{- continue }}
{{- end}}
{{ $deployment := $app_config.harness.deployment }}
{{ $app_name }}:
{{- with $app_config.domain }}
domainname: {{ . }}
{{- end }}
networks:
- ch
{{- with $app_config.image }}
image: {{ . }}
{{- end }}
{{- with $app_config.harness.service.port }}
ports:
- "{{ . }}:{{ $app_config.harness.deployment.port }}"
{{- end}}
deploy:
mode: "replicated"
replicas: {{ $deployment.replicas | default 1 }}
resources:
limits:
cpus: {{ $deployment.resources.limits.cpu | default "50m" }}
memory: {{ trimSuffix "i" $deployment.resources.limits.memory | default "64M" }}
reservations:
cpus: {{ $deployment.resources.requests.cpu | default "25m" }}
memory: {{ trimSuffix "i" $deployment.resources.requests.memory | default "32M" }}
environment:
- CH_CURRENT_APP_NAME={{ $app_name | quote }}

{{- range $.Values.env }}
- {{ .name }}={{ .value | quote }}
{{- end }}
{{- /*{{- range $.Values.env }}
- {{ .name }}={{ .value | quote }}
{{- end }} */}}
{{- range $app_config.harness.env }}
- {{ .name }}={{ .value | quote }}
{{- end }}
{{- with $app_config.harness.dependencies.soft }}
# links:
# {{- range . }}
# - {{ . }}
# {{- end }}
{{- end }}
{{- with $app_config.harness.dependencies.hard }}
depends_on:
{{- range . }}
- {{ . }}
{{- end }}
{{- end }}
{{- if or $deployment.volume $app_config.harness.resources }}
volumes:
{{- with $deployment.volume }}
- type: volume
source: {{ .name }}
target: {{ .mountpath }}
{{- end}}
{{- with $app_config.harness.resources }}
{{- range .}}
- type: bind
source: compose/resources/{{ $app_name }}/{{.src }}
target: {{ .dst }}
{{- end }}
{{- end}}
{{- end }}
{{- end }}

traefik:
image: "traefik:v2.2"
container_name: "traefik"
networks:
- ch
command:
- "--log.level=INFO"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--providers.file.directory=/etc/traefik/dynamic_conf"
ports:
- "80:80"
- "443:443"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./certs/:/certs/:ro"
- "./traefik.yaml:/etc/traefik/dynamic_conf/conf.yml:ro"

networks:
ch:
name: ch_network

volumes: # this inclusion needs to be conditional
{{- range $app_name, $app_config := .Values.apps }}
{{- with $app_config.harness.deployment.volume }}
{{ .name }}:
{{- end }}
{{- end }}
79 changes: 79 additions & 0 deletions deployment-configuration/compose/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# -- If set to true, local DNS mapping is added to pods.
filippomc marked this conversation as resolved.
Show resolved Hide resolved
local: false
# -- Enables/disables Gatekeeper.
secured_gatekeepers: true
# -- The root domain.
domain: ${{DOMAIN}}
# -- The K8s namespace.
namespace: ch
# -- Name of mainapp, routes incoming traffic of root `domaim` to this app.
mainapp: accounts
registry:
# -- The docker registry.
name: "localhost:5000"
# -- Optional secret used for pulling from docker registry.
secret:
# -- Docker tag used to pull images.
tag: latest
# -- List of applications.
# @default -- Will be filled automatically.
apps: {}
env:
# -- Cloud Harness version
- name: CH_VERSION
value: 0.0.1
# -- Cloud harness chart version
- name: CH_CHART_VERSION
value: 0.0.1
privenv:
# -- Defines a secret as private environment variable that is injected in containers.
- name: CH_SECRET
value: In God we trust; all others must bring data. ― W. Edwards Deming
ingress:
# -- Flag to enable/disalbe ingress controller.
enabled: true
# -- K8s Name of ingress.
name: cloudharness-ingress
# -- Enables/disables SSL redirect.
ssl_redirect: true
letsencrypt:
# -- Email for letsencrypt.
email: filippo@metacell.us
backup:
# -- Flag to enable/disable backups.
active: false
# -- Number of days to keep backups.
keep_days: "7"
# -- Number of weeks to keep backups.
keep_weeks: "4"
# -- Number of months to keep backups.
keep_months: "6"
# -- Schedule as cronjob expression.
schedule: "*/5 * * * *"
# -- The file suffix added to backup files.
suffix: ".gz"
# -- The volume size for backups (all backups share the same volume)
volumesize: "2Gi"
# -- Target directory of backups, the mount point of the persistent volume.
dir: "/backups"
resources:
requests:
# -- K8s memory resource definition.
memory: "32Mi"
# -- K8s cpu resource definition.
cpu: "25m"
limits:
# -- K8s memory resource definition.
memory: "64Mi"
# -- K8s cpu resource definition.
cpu: "50m"
proxy:
timeout:
# -- Timeout for proxy connections in seconds.
send: 60
# -- Timeout for proxy responses in seconds.
read: 60
keepalive: 60
payload:
# -- Maximum size of payload in MB
max: 250
Loading