Skip to content

Commit

Permalink
Release/0.2.0 (#117)
Browse files Browse the repository at this point in the history
* Adding logo file

* Update README.md

* #4 Fix tls certificates for minikubes

* chore: added themes/custom folder + how to README.md

* fix: fixes #11 problem of mounting vol1, vol2 in minikube

* fix: fixes #13, set theme to keycloak because there is no custom theme. The custom theme setting causes the null pointer exception

* fix: fixes #14, username is a required field which (for now) needs to be entered on the user form

* chore: use docker commands to copy and retrieve files instead of minikube shared folders

* chore: added frontend client for the frontend app

* chore: disable verification email

* chore: rename frontend to portal

* chore: renamed frontend to portal

* Small fix and doc improvements

* Fix realm address

* fix: use output_path for generating the certificates

* chore: removed the www kc client and made client web-client public

* #25 new values specification implemented

* #25 fixes and refactoring

* #28 Add JupyterHub

* #28 Service/ingress configuration

* #28 chore name change

* #28 fixed references

* #25 chore fix autoservice to use new config names

* Fix auto-service

* Fix auto-service

* Fixed realm conf error

* #30 Hook into jupyterhub spawner pod definition

* Fix base images path on codefresh

* Improve codefresh script templates

* #32 Fixed secret assignment to built images

* Fix gatekeeper image version

* Fix jupyterhub pull secret

* Add secret setting to jupyterhub

* #34 new sentry service for capturing application events (logs, errors etc) 

#34 new sentry service for capturing application events (logs, errors etc) 

* add: new sentry service for capturing application events (logs, errors etc)

4 new pods:
sentry: main sentry web app
sentry-postgres-svr: postgres database for sentry
sentry-redis-svr: redis server for sentry
sentry-postfix-svr: mail MTA server for sentry (postfix)

for more info about sentry see https://sentry.io/

* fix sentry docker version to 9.1.2

* fix: fixate docker redis, postgres and postfix to latest

* chore: removed Dockerfiles for sentry postgres, redis and postix

see feature/sentry-server

added the docker image names and tags for sentry postgres, postfix and redis to values.yaml
removed the variable tag from sentry postgres, postfix, redis images so we pull always latest
added tag 9.1.2 to sentry image

* #37 Add sentry application and service

* add: new sentry service for capturing application events (logs, errors etc)

4 new pods:
sentry: main sentry web app
sentry-postgres-svr: postgres database for sentry
sentry-redis-svr: redis server for sentry
sentry-postfix-svr: mail MTA server for sentry (postfix)

for more info about sentry see https://sentry.io/

* Create Python api to retrieve infrastructure information

see #33
chore: added conf object to cloudharness utils, reworked env functions to use the conf object
  the new conf object is a simple wrapper around the allvalues.yaml
  it provides one function: get_application_by_filter
  returns app objects filtered by the filter
  example:
    conf.get_application_by_filter(harness__deployment__auto=True)
add: mount helm values.yaml to autodeploy applications
  add config map to transform the values into allvalues.yaml file
  mount the allvalues.yaml to the pod in /opt/cloudharness/resources/allvalues.yaml
add: added interface to access internal cloudharness allvalues dictionary

* chore: create new micro chservice api service for CH services

see osb #35
added sentry service for getting sentry DSN for applications
when an application is not yet know in Sentry it will be created
for convenience added cloud harness python library module function
utils.sentry.get_dsn
Usage examples:
  from cloudharness.sentry import get_dsn
  dsn = get_dsn('workspaces')

* small change

* removed bootstrap.sh.tar

* chore: refactoring utils/config.py

see #33
refactor so we have classmethods instead of instance methods
make CloudharnessConfig class public instead of private

* chore: added class comment to CloudharnessConfig class

see #33

* chore: add test if app has the harness.sentry setting set to True, if not or False then return '' indicating no Sentry DSN

see #35 osb

* fix: remove duplicate deployment element

* chore: refactor sentry to errormonitor, chservice to common, created cloudharness init, created sentry init to cloudharness init

see osb #35

* fix: fix reading sentry setting location, should be app.sentry instead of app.harness.sentry

see osb v2 #35

* fix: generate a comlete dsn instead of only the dsn public key

see osb v2 #35

* #37 Common service fixes

* add: new sentry service for capturing application events (logs, errors etc)

4 new pods:
sentry: main sentry web app
sentry-postgres-svr: postgres database for sentry
sentry-redis-svr: redis server for sentry
sentry-postfix-svr: mail MTA server for sentry (postfix)

for more info about sentry see https://sentry.io/

* Create Python api to retrieve infrastructure information

see #33
chore: added conf object to cloudharness utils, reworked env functions to use the conf object
  the new conf object is a simple wrapper around the allvalues.yaml
  it provides one function: get_application_by_filter
  returns app objects filtered by the filter
  example:
    conf.get_application_by_filter(harness__deployment__auto=True)
add: mount helm values.yaml to autodeploy applications
  add config map to transform the values into allvalues.yaml file
  mount the allvalues.yaml to the pod in /opt/cloudharness/resources/allvalues.yaml
add: added interface to access internal cloudharness allvalues dictionary

* chore: create new micro chservice api service for CH services

see osb #35
added sentry service for getting sentry DSN for applications
when an application is not yet know in Sentry it will be created
for convenience added cloud harness python library module function
utils.sentry.get_dsn
Usage examples:
  from cloudharness.sentry import get_dsn
  dsn = get_dsn('workspaces')

* small change

* removed bootstrap.sh.tar

* chore: refactoring utils/config.py

see #33
refactor so we have classmethods instead of instance methods
make CloudharnessConfig class public instead of private

* chore: added class comment to CloudharnessConfig class

see #33

* chore: add test if app has the harness.sentry setting set to True, if not or False then return '' indicating no Sentry DSN

see #35 osb

* fix: remove duplicate deployment element

* chore: refactor sentry to errormonitor, chservice to common, created cloudharness init, created sentry init to cloudharness init

see osb #35

* fix: fix reading sentry setting location, should be app.sentry instead of app.harness.sentry

see osb v2 #35

* fix: generate a comlete dsn instead of only the dsn public key

see osb v2 #35

* fix: add cors for common

see osb v2 #35

* Feature/37 Add sentry test (#42)

* #37 Update cli
* #37 added sentry test
* #37 added sentry api test

* #33 High level configuration api (#41)

* #33 High level application configuration api
* #33 Use new api on Sentry controller

* fix: changed some CH settings for sentry and sentry postgres

* osb # 46

* fix: add try catch around sentry init to overcome crashing of applications when sentry is not up/present

* see osb #46

* fix: changed postgres mount persistent volume to mount on /opt/data and initdb in /opt/data/pgdata because on GC the mounted volume isn't empty

* see osb #46

* Remove gatekeeper on samples

* Fix argo workflows python lib

* Improve command line tools and related documentation.

* #45 add persistent volume option to auto deploy apps (#44)

* chore: add persistent volume option to auto deploy apps

* chore: add support for persistent volume to auto deployment

* #46 increase proxy-body-size of Ingress server to 10M

* Generate client fix

* #50 Add parameter to specify included applications (#51)

* #50 Add parameter to specify included applications
* #50 Align include criterion with build

* #52 add namespace and domain to build args so it can be used during building of the images

* #52 fix typo

* #43 #57 JupyterHub custom authenticator and volume manager (#56)

* #43 connect jupyterhub to cloudharness keycloak

* #43 chore: some code cleanup and added namedServerLimitPerUser=10 config to values.yaml

* #57 chore: moved kubernetes pvc code from osb to ch

* #57 chore: added use of kubernetes config when not in a cluster

* #57 chore: added application hook

* #57 fix: when no user is provided logout user

* #57 chore: rm keycloakauthenticator

* #57 chore: change PVC accessmode to ReadWriteOnce

* #57 chore: removed LICENSE from chauthenticator

* #57 chore: removed ToDo flag and added comment on applicationHook usage

* #57 chore: moved volume manager from OSBv2 to CloudHarness

* chore: renamed some OSB references to CH

* #58 fix: changed keyUsage to nonRepudiation, digitalSignature, keyEncipherment (#59)

* #55 Application overriding introduced

* #54 Remove reference to other projects

* #55 Implement directory merge to build images

* #55 Use merge strategy

* #55 Instantiate docker client only on run

* #55 Fix merging

* #55 Remove / from registry in helm chart

* #55 Fix ingress for domain

* #55 Fix auto-gatekeeper

* #55 Fix dpaths typo in directorymerge operation

* #55 Fix dpaths typo in directorymerge operation

* #55 Debug CF pipeline

* #55 Test if CF is caching docker layers

* #54 Allow private keys as build-args

* #55 Polish PR

* #55 Polish PR

Co-authored-by: rodriguez-facundo <[email protected]>
Co-authored-by: Filippo Ledda <[email protected]>

* chore: added application specific secrets to deploy/values.yaml and cloudharness utilities

e.g. config
harness:
  ...
  secrets:
    - name: secret_1
      value: value_1
    - name: secret_2
      value: value_2
  ...

e.g. usage
from cloudharness.utils.secrets import get_secret

value_1 = get_secret("secret_1")
print(f"Secret 1 value: {value_1}"

* #62 Enable workflow tasks to mount an existing PVC  (merge PR #63)

* #63 chore: add zip extract to download task and use the default CH base image so all CH tools are present in the container

* #62 chore: enable mounting existing PVC in an argo workflow

* #62 fix: remove debug print() statement

* #61 chore: rework so application secrets need to be created on deployment and not on install

* #64 Bind gatekeeper to port 80, update docker image for gatekeeper

* Workflow fixes

* Default registry removed

* #66 fix: added argo-service to proxy to the argo ui and changed the argo port

* #66 chore: removed extra argo service entry and fixed argo values.yaml

* #67 fix: issue with events not working

* #68 fix: issue sentry handler not connecting to common rest api

* #69 chore: added EventClient.async_consume for async consuming events from a topic

* chore: changed loglevel to INFO

* #69 fix: fix issues, added more robust exception handling, added new image notify-queue for doing queue notification

* #70 SSL redirect option added

* #71 fix: replace double or more trialing slashes for registry to only one slash

* #71 fix: replace double or more trialing slashes for registry to only one slash

* Improved deploy robustness on missing applications

* #73 fix: add extra try except and sleep(10) to the thread when the kafka client is disconnected

* #78 Add option to disable tls (#79)

* #78 option to disable tls added
* #78 add tls option to accounts configuration

* #76 chore: added create kafka cluster on init

* Feature/78 (#81)

* #78 option to disable tls added
* #78 revert unrelated change
* #78 add tls option to accounts configuration
* #78 Fix no-http with jupyterhub
* #78 fix set auto gatekeeper secure cookie to false if tls is disabled

* Fix letsencrypt error

* fix letsencrypt

* fix letsencrypt

* #83 removed default reference to accounts application (#84)

* #82 chore: add uri role mapping for auto gk pods to (un)secure uri paths

* #86: added namespace to letsencrypt issuer name

* #86: removed doulbe - from ClusterIssuer name letsencrypt

* #88 add namespace to cluster default zookeeper hosts (#89)

* #90 make ingress listen to the naked domain

* #91 chore: added third parameter (folder) to the download extract task (#92)

* #94 chore: changed ClusterRole to Cluster and added namespace

* #100 fix: issue with filtering on getting applications by boolean filter True/False (#101)

* Feature/96 Support multiple paths in codefresh generate script (#98)

* #96 Support added to multiple directory in codefresh generator

* #102 chore: add get Keycloak user, group and users support to CH (#103)

* #102 chore: add get Keycloak user, group and users support to CH

* #102 fix: added missing HOST variable

* #102 chore: added get_current_user to get the current logged in user

* #97 New directory merge implementation (#104)

* #97 Directory merge do not change current deployment anymore
* #97 Fix build with merge
* #97 Fix build/codefresh with merge
* #97 Add create merge directory if does not exist
* #97 Merge copy preserving file attributes for better caching
* #97 Fix build include regression

* #105 chore: reworked keycloak client and added members to get_groups

* #105 fix: replaced ifn credentials/url by mnp credentials/url

* #105 chore: made get group and get admin client publicable

* #105 chore: added python-keycloak==0.22.0 to setup requirements

* #105 chore: reworked AuthClient to to use instance functions instead of static, added user has client role tests

* #105 fix: replaced wrong values for default realm, host and user in CH keycloak client

* #105 chore: some additions and bugfixes to CH keycloak library

* Feature/110 (#112)

* #110 Copy deployment-configuration with merge
* #111 run codefresh generation with harness-deployment

* Fix build parameter

* fix: replace .get[...] by .get(...)

* #113 chore: add keycloak create client and client role to ch common python lib

* Version update

* #115 chore: add/update/delete kc user attributes

* #115 fix: removed try except for add, update and delete user attributes

* #115 chore: add get users for client with a specific role

Co-authored-by: tarelli <[email protected]>
Co-authored-by: rodriguez-facundo <[email protected]>
Co-authored-by: Zoran Sinnema <[email protected]>
Co-authored-by: rodriguez-facundo <[email protected]>
  • Loading branch information
5 people authored Dec 7, 2020
1 parent 2215470 commit 77d2057
Show file tree
Hide file tree
Showing 252 changed files with 9,167 additions and 2,174 deletions.
21 changes: 20 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
<p align="center">
<img src="https://github.com/MetaCell/cloud-harness/blob/develop/cloudharness.png?raw=true" alt="drawing" width="200"/>
</p>

# CloudHarness
CloudHarness is a base infrastructure facilitator for micro-service based applications deployed on Kubernetes.

Expand All @@ -12,6 +16,16 @@ What building your cluster application with CloudHarness gives to you:
* Submit batch and asynchronous workflows - based on Argo
* Orchestrate Micro-services - based on Kafka

## Command line tools

CloudHarness provides the following command line tools to help application scaffolding and deployment.

* `harness-deployment` - generate the helm chart to deploy on Kubernetes.
* `harness-application` - create a new CloudHarness REST application.
* `harness-codefresh` - generate the Codefresh continuous deployment specification.
* `harness-generate` - generates server and client code for all CloudHarness REST applications.


## Get started

### Prerequisites
Expand Down Expand Up @@ -104,7 +118,12 @@ images are pushed after the build.
Any public registry will work. The suggested way to go is to install a registry on localhost:5000 inside
the kube cluster and push on that registry, also forwarded to localhost.

More info inside `./registry/README.md`.
On minikube can use the registry addon:

`minikube addons enable registry`

Then forward with:
`kubectl port-forward --namespace kube-system $(kubectl get po -n kube-system | grep registry | grep -v proxy | \awk '{print $1;}') 5000:5000`

### Argo installation

Expand Down
2 changes: 1 addition & 1 deletion applications/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Dockerfile

## Define an application without openapi
1. Add the application inside `applications/[APPLICATION_NAME]` with a Docker file in it. The Docker file must inherit
from `r.cfcr.io/tarelli/cloudharness-base` in order to get access to cloudharness libraries.
from `cloudharness-base` in order to get access to cloudharness libraries.
1. Define values.yaml inside the file in order to specify custom values for the application


Expand Down
22 changes: 12 additions & 10 deletions applications/accounts/deploy/resources/realm.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@
"realm": {{ .Values.namespace | quote }},
"enabled": true,
"sslRequired": "external",
"loginTheme": "custom",
"accountTheme": "custom",
"adminTheme": "custom",
"emailTheme": "custom",
"loginTheme": "keycloak",
"accountTheme": "keycloak",
"adminTheme": "keycloak",
"emailTheme": "keycloak",
"registrationAllowed": true,
"registrationEmailAsUsername": true,
"registrationEmailAsUsername": false,
"rememberMe": true,
"verifyEmail": true,
"verifyEmail": false,
"loginWithEmailAllowed": true,
"duplicateEmailsAllowed": false,
"resetPasswordAllowed": true,
Expand Down Expand Up @@ -49,7 +49,7 @@
{
"id": "9a6a2560-c6be-4493-8bd5-3fdc4522d82b",
"clientId": {{ .Values.apps.accounts.client.id | quote }},
"baseUrl": {{ printf "https://%s" .Values.domain | quote }},
"baseUrl": {{ printf "%s://accounts.%s" (ternary "http" "https" (not .Values.tls)) .Values.domain | quote }},
"surrogateAuthRequired": false,
"enabled": true,
"clientAuthenticatorType": "client-secret",
Expand Down Expand Up @@ -84,7 +84,7 @@
{
"id": "111caf43-3d26-484d-8dc9-7fa911ac221c",
"clientId": {{ .Values.apps.accounts.webclient.id | quote }},
"baseUrl": {{ printf "https://%s.%s" .Values.apps.events.subdomain .Values.domain | quote }},
"baseUrl": {{ printf "%s://accounts.%s" (ternary "http" "https" (not .Values.tls)) .Values.domain | quote }},
"surrogateAuthRequired": false,
"enabled": true,
"clientAuthenticatorType": "client-secret",
Expand All @@ -93,10 +93,12 @@
"*"
],
"webOrigins": [
"*"
"*",
"+"
],
"standardFlowEnabled": true,
"directAccessGrantsEnabled": true,
"publicClient": true,
"protocol": "openid-connect",
"fullScopeAllowed": true,
"defaultClientScopes": [
Expand Down Expand Up @@ -623,5 +625,5 @@
}
}
],
"keycloakVersion": "6.0.1"
"keycloakVersion": "9.0.2"
}
37 changes: 0 additions & 37 deletions applications/accounts/keycloak-gatekeeper/Dockerfile

This file was deleted.

3 changes: 3 additions & 0 deletions applications/accounts/themes/custom/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Custom theme for Keycloak

For custom theme development see: [Keycloak themes development](https://www.keycloak.org/docs/latest/server_development/index.html#_themes)
17 changes: 10 additions & 7 deletions applications/argo/deploy/values.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
enabled: true
name: argo-server
subdomain: argo
port: 2746
harness:
subdomain: argo
secured: true
name: argo-proxy1
service:
port: 80
auto: false
name: argo-ui

serviceaccount: argo-workflows
namespace: argo
namespace: argo.svc.cluster.local
workflows_namespace: argo-workflows
secureme: true
autoservice: false
3 changes: 3 additions & 0 deletions applications/common/api/config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"packageName": "common"
}
35 changes: 35 additions & 0 deletions applications/common/api/openapi.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
openapi: 3.0.0
info:
description: Cloud Harness Platform - Reference CH service API
license:
name: UNLICENSED
title: CH common service API
version: 0.1.0
servers:
- description: SwaggerHub API Auto Mocking
url: /api
tags:
- description: Sentry
name: Sentry
paths:
/sentry/getdsn/{appname}:
parameters:
- in: path
name: appname
schema:
type: string
required: true
get:
tags:
- Sentry
description: Gets the Sentry DSN for a given application
operationId: getdsn
responses:
'200':
description: Sentry DSN for the given application
content:
application/json:
schema:
type: string
summary: Gets the Sentry DSN for a given application
x-openapi-router-controller: common.controllers.sentry_controller
19 changes: 19 additions & 0 deletions applications/common/deploy/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
harness:
subdomain: common
secured: false
service:
auto: true
port: 8080
name: common
deployment:
auto: true
name: common
port: 8080
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 256Mi
cpu: 200m

File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,4 @@
# Then explicitly reverse the ignore rule for a single file:
#!docs/README.md

Dockerfile
*/controllers/*
*/models/*
*/controllers/*
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
ARG REGISTRY=r.cfcr.io/tarelli/
ARG REGISTRY
ARG TAG=latest
FROM ${REGISTRY}cloudharness-base:${TAG}

RUN apk update && apk add postgresql-dev gcc python3-dev musl-dev
#RUN apk add psycopg2 libpq-dev python-dev

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

Expand All @@ -13,6 +16,9 @@ COPY . /usr/src/app

EXPOSE 8080

ENV FLASK_ENV=production
ENV APP_SETTINGS=common.config.ProductionConfig

ENTRYPOINT ["python3"]

CMD ["-m", "workflows_api"]
CMD ["-m", "common"]
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,19 @@ To run the server, please execute the following from the root directory:

```
pip3 install -r requirements.txt
python3 -m workflows_api
python3 -m openapi_server
```

and open your browser to here:

```
http://localhost:8080/ui/
http://localhost:8080/api/ui/
```

Your OpenAPI definition lives here:

```
http://localhost:8080/openapi.json
http://localhost:8080/api/openapi.json
```

To launch the integration tests, use tox:
Expand All @@ -42,8 +42,8 @@ To run the server on a Docker container, please execute the following from the r

```bash
# building the image
docker build -t workflows_api .
docker build -t openapi_server .

# starting up a container
docker run -p 8080:8080 workflows_api
docker run -p 8080:8080 openapi_server
```
23 changes: 23 additions & 0 deletions applications/common/server/common/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import os

from flask import Flask
from flask_cors import CORS

import connexion

from common import encoder

def main():
app = connexion.App(__name__, specification_dir='./openapi/')
app.app.config.from_object(os.environ['APP_SETTINGS'])
app.app.json_encoder = encoder.JSONEncoder
app.add_api('openapi.yaml',
arguments={'title': 'CH service API'},
pythonic_params=True)
from .repository.db import open_db
open_db(app)
cors = CORS(app.app, resources={r"/api/*": {"origins": "*"}})
app.run(port=8080)

if __name__ == '__main__':
main()
41 changes: 41 additions & 0 deletions applications/common/server/common/config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
import os
import logging

from cloudharness.utils.config import CloudharnessConfig as conf

basedir = os.path.abspath(os.path.dirname(__file__))


class Config(object):
DEBUG = False
TESTING = False
CSRF_ENABLED = True
SQLALCHEMY_TRACK_MODIFICATIONS = False
SECRET_KEY = 'this-really-needs-to-be-changed'
SENTRY_POSTGRES_APP = None
SENTRY_APP = None
try:
SENTRY_POSTGRES_APP = conf.get_application_by_filter(name='sentry')[0].postgres
SENTRY_APP = conf.get_application_by_filter(name='sentry')[0].name
SQLALCHEMY_DATABASE_URI = f'postgresql+psycopg2://{SENTRY_POSTGRES_APP.user}:{SENTRY_POSTGRES_APP.password}@{SENTRY_POSTGRES_APP.name}:{SENTRY_POSTGRES_APP.port}/{SENTRY_POSTGRES_APP.initialdb}'
except:
logging.error("Cannot configure SENTRY")



class ProductionConfig(Config):
DEBUG = False


class StagingConfig(Config):
DEVELOPMENT = True
DEBUG = True


class DevelopmentConfig(Config):
DEVELOPMENT = True
DEBUG = True


class TestingConfig(Config):
TESTING = True
Loading

0 comments on commit 77d2057

Please sign in to comment.