-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CH-100 Add preliminary docker-compose support #735
Merged
Merged
Changes from 1 commit
Commits
Show all changes
102 commits
Select commit
Hold shift + click to select a range
d25230f
CH-100 Add first code to call a dedicated docker-compose generation
aranega 6f306a1
Squashed commit of the following:
aranega 6bbae19
CH-100 Add first skaffold dedicated generation for docker compose
aranega 5287545
CH-100 Make skaffold script a little bit more generic (ugly)
aranega 9f75c9c
CH-100 Fix issue with entrypoint
aranega c159a4f
CH-100 Remove generation of chart files for docker-compose
aranega 02bd318
CH-100 Rollback default per-app helm template directory
aranega 1a57064
CH-100 Add subdomain configuration
aranega b97c19c
CH-100 Fix bad "links" generation
aranega 85dcfd9
CH-100 Add support for aliases and service links
aranega 1ac8226
CH-100 Add first support for auto databases
aranega 56e22b7
CH-100 Add finer grain port handling
aranega 336b558
CH-100 Change way port is exposed to outside world
aranega a004ffb
CH-100 Fix issue with env var quoting
aranega f73108c
CH-100 Add special behavior to produce allvalues.yaml
aranega 2370b20
CH-100 Add actual docker-compose.yaml generation in harness-deployment
aranega 5ed630d
CH-100 Add first dedicated templates for postgres
aranega 111a4f3
CH-100 Add volumes for db
aranega ce4596a
CH-100 Add first template for events
aranega 3868b56
CH-100 Refactor templates for the db
aranega a0476d6
CH-100 Add /etc/hosts generation
aranega bd3e423
CH-100 Add specific varenvs by container
aranega 8851c48
CH-100 Rename some templates
aranega ced7cd2
CH-100 Add first traeffik configuration
aranega f7ebeb0
CH-100 Fix bad indentation in docker-compose.yaml
aranega 8fdcba6
CH-100 Remove exposition of ports
aranega 428a619
CH-100 Add post-process mechanism to generate files
aranega 9d6c8b2
CH-100 Add new templates
aranega ae7cf19
CH-100 Add new templates (not modified yet)
aranega 7acc7f5
CH-100 Add generation of resources files
aranega 571c2ab
CH-100 Add dependency between service and db
aranega 9c8c19e
CH-100 Add secret handling
aranega f6d0c67
CH-100 Remove argo from dependencies
aranega 4fd6dee
CH-100 Change from pyyaml to ruamel
aranega 86c0e4f
CH-100 Remove tmp "events" from deps
aranega a5909ce
CH-100 Add back sc.yaml
aranega 4379252
CH-100 Add first handling of NFS volume
aranega 46ee140
CH-100 Add volumes conditionnally
aranega e135c6a
CH-100 Add first tutorial about docker compose
aranega 0f70146
CH-100 Add first real config for gatekeeper
aranega 6bbef64
CH-100 Add reverse proxy config for gatekeeper
aranega 3b437cf
CH-100 Fix path to generated resources
aranega 1135b99
CH-100 Fix bad GK configuration
aranega b23f03b
CH-100 Adapt gk configuration
aranega 361ca46
CH-100 Fix gk configuration
aranega 74d074e
CH-100 Comment unused "ports"
aranega fec25be
CH-100 Fix issue with service names in dependencies
aranega 0db12d8
CH-100 Fix issue with db-volumes
aranega f8d4c84
CH-100 Fix health check for account services
aranega 4121180
CH-100 Fix issue with dependencies
aranega d81abc2
Merge branch 'develop' of github.com:MetaCell/cloud-harness into feat…
aranega 5bf5532
CH-100 Change tagPolicy for docker-compose target
aranega 319352b
CH-100 Add first information for the docker compose target
aranega 7ec48e1
CH-100 Update documentation
aranega d85389a
Merge branch 'develop' into feature/CH-100-merge
aranega 7edfcc4
#CH-100 improve documentation
filippomc e5d56b7
CH-100 First refactoring
aranega 368725a
CH-100 Clean imports/comments
aranega a4cd813
CH-100 Refactor helm test
aranega 1d63bc4
CH-100 Add tests for docker compose target
aranega b125dfc
CH-100 Add tests for docker compose
aranega 891b237
Merge branch 'feature/CH-100' of github.com:MetaCell/cloud-harness in…
aranega 2c7e264
CH-100 Fix type hinting for Python 3.9
aranega 866dee9
CH-100 Add conditional test for docker compose if "helm" is installed
aranega f226829
CH-100 Update scripts
aranega ac4b863
CH-100 Fix imports of harness-deployment
aranega 9b4dcaa
CH-100 Fix issue with TAG policy in skaffold for docker compose
aranega 10f2c0f
CH-100 Update configuration for db in docker compose
aranega 83d9b8a
CH-100 Add support to link databases with service in docker compose
aranega e5361ca
CH-100 Change location of traefik config files
aranega 4c5d369
CH-100 Add specific option to expose locally a DB
aranega 6cf0a83
Merge branch 'develop' of github.com:MetaCell/cloud-harness into feat…
filippomc fba11c6
Merge branch 'develop' of github.com:MetaCell/cloud-harness into feat…
filippomc 4f5f34c
CH-110 fix dc resources mapping when not numbers
filippomc bbc57d2
chore: remove unneded dependencies
filippomc 2a87ba5
CH-110 small fixes + tests
filippomc 5adab28
CH-110 chore: ignore generated compose file
filippomc c10719e
CH-126 update python client generation
filippomc b344129
CH-110 improve output on error
filippomc 1ddd023
CH-126 improve code generation
filippomc 97ff173
CH-110 improve output on error
filippomc a5477a9
CH-110 fix tests for docker-compose
filippomc e5fb656
Merge branch 'develop' of github.com:MetaCell/cloud-harness into feat…
filippomc 501ad71
Merge branch 'feature/CH-147-lint' of github.com:MetaCell/cloud-harne…
filippomc f69b4dd
Merge branch 'develop' of github.com:MetaCell/cloud-harness into feat…
filippomc 8bdb619
CH-66 change secrets in configmap
filippomc ea20917
Fix merge issues
filippomc 4fe5c4c
Remove wrong dependency
filippomc b9c3979
CH-147 fix linting issue
filippomc e838008
CH-17 fix some dependencies
filippomc 4a04180
CH-17 fix python backend debug
filippomc 5a4700f
Update test CI/CD
filippomc 65bcc2b
CH-17 fix build issue
filippomc 1562e95
CH-17 fix broken API tests
filippomc 0f88e07
Improve testing docs
filippomc 2535a16
Improve cookie auth automations
filippomc 18c47cd
Lint fixes
filippomc 0eae79b
Lint fixes
filippomc 3529200
CH-110 small changes to docker compose generation
filippomc 08b9645
CH-100 refactoring, small fixes
filippomc 7eb8ca3
Linting fix
filippomc 8bb9a1b
Linting fix
filippomc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next
Next commit
CH-100 Add first code to call a dedicated docker-compose generation
- Loading branch information
commit d25230ff5144cfead4bf98eb343ddd684dfa654a
There are no files selected for viewing
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
# Patterns to ignore when building packages. | ||
# This supports shell glob matching, relative path matching, and | ||
# negation (prefixed with !). Only one pattern per line. | ||
.DS_Store | ||
# Common VCS dirs | ||
.git/ | ||
.gitignore | ||
.bzr/ | ||
.bzrignore | ||
.hg/ | ||
.hgignore | ||
.svn/ | ||
# Common backup files | ||
*.swp | ||
*.bak | ||
*.tmp | ||
*~ | ||
# Various IDEs | ||
.project | ||
.idea/ | ||
*.tmproj | ||
.vscode/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
apiVersion: v1 | ||
appVersion: "0.0.1" | ||
description: CloudHarness Helm Chart | ||
name: cloudharness | ||
version: 0.0.1 | ||
maintainers: | ||
- name: Filippo Ledda | ||
email: filippo@metacell.us | ||
- name: Zoran Sinnema | ||
email: zoran@metacell.us |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# CloudHarness Helm chart: deploy CloudHarness to k8s | ||
|
||
Helm is used to define the CloudHarness deployment on Kubernetes. For further information about Helm, see https://helm.sh. | ||
|
103 changes: 103 additions & 0 deletions
103
deployment-configuration/compose/templates/auto-compose.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,103 @@ | ||
version: '3.7' | ||
|
||
services: | ||
{{- range $app_name, $app_config := .Values.apps }} | ||
{{- if has $app_name (list "argo" "nfsserver" "workflows" "events" ) }} {{- /* We deactivate generation for some services */}} | ||
{{- continue }} | ||
{{- end}} | ||
{{ $deployment := $app_config.harness.deployment }} | ||
{{ $app_name }}: | ||
{{- with $app_config.domain }} | ||
domainname: {{ . }} | ||
{{- end }} | ||
networks: | ||
- ch | ||
{{- with $app_config.image }} | ||
image: {{ . }} | ||
{{- end }} | ||
{{- with $app_config.harness.service.port }} | ||
ports: | ||
- "{{ . }}:{{ $app_config.harness.deployment.port }}" | ||
{{- end}} | ||
deploy: | ||
mode: "replicated" | ||
replicas: {{ $deployment.replicas | default 1 }} | ||
resources: | ||
limits: | ||
cpus: {{ $deployment.resources.limits.cpu | default "50m" }} | ||
memory: {{ trimSuffix "i" $deployment.resources.limits.memory | default "64M" }} | ||
reservations: | ||
cpus: {{ $deployment.resources.requests.cpu | default "25m" }} | ||
memory: {{ trimSuffix "i" $deployment.resources.requests.memory | default "32M" }} | ||
environment: | ||
- CH_CURRENT_APP_NAME={{ $app_name | quote }} | ||
|
||
{{- range $.Values.env }} | ||
- {{ .name }}={{ .value | quote }} | ||
{{- end }} | ||
{{- /*{{- range $.Values.env }} | ||
- {{ .name }}={{ .value | quote }} | ||
{{- end }} */}} | ||
{{- range $app_config.harness.env }} | ||
- {{ .name }}={{ .value | quote }} | ||
{{- end }} | ||
{{- with $app_config.harness.dependencies.soft }} | ||
# links: | ||
# {{- range . }} | ||
# - {{ . }} | ||
# {{- end }} | ||
{{- end }} | ||
{{- with $app_config.harness.dependencies.hard }} | ||
depends_on: | ||
{{- range . }} | ||
- {{ . }} | ||
{{- end }} | ||
{{- end }} | ||
{{- if or $deployment.volume $app_config.harness.resources }} | ||
volumes: | ||
{{- with $deployment.volume }} | ||
- type: volume | ||
source: {{ .name }} | ||
target: {{ .mountpath }} | ||
{{- end}} | ||
{{- with $app_config.harness.resources }} | ||
{{- range .}} | ||
- type: bind | ||
source: compose/resources/{{ $app_name }}/{{.src }} | ||
target: {{ .dst }} | ||
{{- end }} | ||
{{- end}} | ||
{{- end }} | ||
{{- end }} | ||
|
||
traefik: | ||
image: "traefik:v2.2" | ||
container_name: "traefik" | ||
networks: | ||
- ch | ||
command: | ||
- "--log.level=INFO" | ||
- "--api.insecure=true" | ||
- "--providers.docker=true" | ||
- "--providers.docker.exposedbydefault=false" | ||
- "--entrypoints.web.address=:80" | ||
- "--entrypoints.websecure.address=:443" | ||
- "--providers.file.directory=/etc/traefik/dynamic_conf" | ||
ports: | ||
- "80:80" | ||
- "443:443" | ||
volumes: | ||
- "/var/run/docker.sock:/var/run/docker.sock:ro" | ||
- "./certs/:/certs/:ro" | ||
- "./traefik.yaml:/etc/traefik/dynamic_conf/conf.yml:ro" | ||
|
||
networks: | ||
ch: | ||
name: ch_network | ||
|
||
volumes: # this inclusion needs to be conditional | ||
{{- range $app_name, $app_config := .Values.apps }} | ||
{{- with $app_config.harness.deployment.volume }} | ||
{{ .name }}: | ||
{{- end }} | ||
{{- end }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,79 @@ | ||
# -- If set to true, local DNS mapping is added to pods. | ||
filippomc marked this conversation as resolved.
Show resolved
Hide resolved
|
||
local: false | ||
# -- Enables/disables Gatekeeper. | ||
secured_gatekeepers: true | ||
# -- The root domain. | ||
domain: ${{DOMAIN}} | ||
# -- The K8s namespace. | ||
namespace: ch | ||
# -- Name of mainapp, routes incoming traffic of root `domaim` to this app. | ||
mainapp: accounts | ||
registry: | ||
# -- The docker registry. | ||
name: "localhost:5000" | ||
# -- Optional secret used for pulling from docker registry. | ||
secret: | ||
# -- Docker tag used to pull images. | ||
tag: latest | ||
# -- List of applications. | ||
# @default -- Will be filled automatically. | ||
apps: {} | ||
env: | ||
# -- Cloud Harness version | ||
- name: CH_VERSION | ||
value: 0.0.1 | ||
# -- Cloud harness chart version | ||
- name: CH_CHART_VERSION | ||
value: 0.0.1 | ||
privenv: | ||
# -- Defines a secret as private environment variable that is injected in containers. | ||
- name: CH_SECRET | ||
value: In God we trust; all others must bring data. ― W. Edwards Deming | ||
ingress: | ||
# -- Flag to enable/disalbe ingress controller. | ||
enabled: true | ||
# -- K8s Name of ingress. | ||
name: cloudharness-ingress | ||
# -- Enables/disables SSL redirect. | ||
ssl_redirect: true | ||
letsencrypt: | ||
# -- Email for letsencrypt. | ||
email: filippo@metacell.us | ||
backup: | ||
# -- Flag to enable/disable backups. | ||
active: false | ||
# -- Number of days to keep backups. | ||
keep_days: "7" | ||
# -- Number of weeks to keep backups. | ||
keep_weeks: "4" | ||
# -- Number of months to keep backups. | ||
keep_months: "6" | ||
# -- Schedule as cronjob expression. | ||
schedule: "*/5 * * * *" | ||
# -- The file suffix added to backup files. | ||
suffix: ".gz" | ||
# -- The volume size for backups (all backups share the same volume) | ||
volumesize: "2Gi" | ||
# -- Target directory of backups, the mount point of the persistent volume. | ||
dir: "/backups" | ||
resources: | ||
requests: | ||
# -- K8s memory resource definition. | ||
memory: "32Mi" | ||
# -- K8s cpu resource definition. | ||
cpu: "25m" | ||
limits: | ||
# -- K8s memory resource definition. | ||
memory: "64Mi" | ||
# -- K8s cpu resource definition. | ||
cpu: "50m" | ||
proxy: | ||
timeout: | ||
# -- Timeout for proxy connections in seconds. | ||
send: 60 | ||
# -- Timeout for proxy responses in seconds. | ||
read: 60 | ||
keepalive: 60 | ||
payload: | ||
# -- Maximum size of payload in MB | ||
max: 250 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same file used on Kubernetes, do we need it to be duplicated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see your point, but trying to share the files between the helm configuration files and the docker compose one will only make things more obscure to understand.
Those two files (this one and the
values.yaml
) that are in thedeployment-configuration
needs to be duplicated in a way. If not, it means that there is a weird behavior that we should implement saying that the content of those files that are required for the template generation to work, need to be took from thehelm
folder for some reasons. It also means that we cannot rely on the current mechanism which copies files for a target from the corresponding folder, but instead, we would have a mix between "I pick files from the dedicatedcompose
folder, but sometimes no, I need to pick them in thehelm
folder", which makes the generator more complicated and the logic of the configuration file generation more obscure.A lot of files appears to be duplicated, but they are here to support the simplicity of "when I generate a target, the files from the corresponding target folder are copied" instead of having a mix of "sometimes its took from this folder, sometimes no", which would also require to implement a mechanism to check if the files that needs to be picked from the
helm
folder needs to be copied only if it doesn't exists in thecompose
folder to support configuration overriding (or other target folder if we support others in the future).Another solution would be to share those files in a
common
orshared-configuration
folder which would require to implement the same logic described above, but with another folder. I would vote against it as it makes the whole logic more complicated to understand, makes the generator more complex than just "copy files from the corresponding folder" and breaks, in a way, the isolation and independence of each target, to avoid having two files that looks the same. Why make it complicated when we can keep it simple and stupid?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see your point the reason to avoid duplication. I'm removing the irrelevant configuration from compose's values.yaml