CloudHarness v0.2.0

Release/0.2.0 (#117)

* Adding logo file

* Update README.md

* #4 Fix tls certificates for minikubes

* chore: added themes/custom folder + how to README.md

* fix: fixes #11 problem of mounting vol1, vol2 in minikube

* fix: fixes #13, set theme to keycloak because there is no custom theme. The custom theme setting causes the null pointer exception

* fix: fixes #14, username is a required field which (for now) needs to be entered on the user form

* chore: use docker commands to copy and retrieve files instead of minikube shared folders

* chore: added frontend client for the frontend app

* chore: disable verification email

* chore: rename frontend to portal

* chore: renamed frontend to portal

* Small fix and doc improvements

* Fix realm address

* fix: use output_path for generating the certificates

* chore: removed the www kc client and made client web-client public

* #25 new values specification implemented

* #25 fixes and refactoring

* #28 Add JupyterHub

* #28 Service/ingress configuration

* #28 chore name change

* #28 fixed references

* #25 chore fix autoservice to use new config names

* Fix auto-service

* Fix auto-service

* Fixed realm conf error

* #30 Hook into jupyterhub spawner pod definition

* Fix base images path on codefresh

* Improve codefresh script templates

* #32 Fixed secret assignment to built images

* Fix gatekeeper image version

* Fix jupyterhub pull secret

* Add secret setting to jupyterhub

* #34 new sentry service for capturing application events (logs, errors etc) 

#34 new sentry service for capturing application events (logs, errors etc) 

* add: new sentry service for capturing application events (logs, errors etc)

4 new pods:
sentry: main sentry web app
sentry-postgres-svr: postgres database for sentry
sentry-redis-svr: redis server for sentry
sentry-postfix-svr: mail MTA server for sentry (postfix)

for more info about sentry see https://sentry.io/

* fix sentry docker version to 9.1.2

* fix: fixate docker redis, postgres and postfix to latest

* chore: removed Dockerfiles for sentry postgres, redis and postix

see feature/sentry-server

added the docker image names and tags for sentry postgres, postfix and redis to values.yaml
removed the variable tag from sentry postgres, postfix, redis images so we pull always latest
added tag 9.1.2 to sentry image

* #37 Add sentry application and service

* add: new sentry service for capturing application events (logs, errors etc)

4 new pods:
sentry: main sentry web app
sentry-postgres-svr: postgres database for sentry
sentry-redis-svr: redis server for sentry
sentry-postfix-svr: mail MTA server for sentry (postfix)

for more info about sentry see https://sentry.io/

* Create Python api to retrieve infrastructure information

see #33
chore: added conf object to cloudharness utils, reworked env functions to use the conf object
  the new conf object is a simple wrapper around the allvalues.yaml
  it provides one function: get_application_by_filter
  returns app objects filtered by the filter
  example:
    conf.get_application_by_filter(harness__deployment__auto=True)
add: mount helm values.yaml to autodeploy applications
  add config map to transform the values into allvalues.yaml file
  mount the allvalues.yaml to the pod in /opt/cloudharness/resources/allvalues.yaml
add: added interface to access internal cloudharness allvalues dictionary

* chore: create new micro chservice api service for CH services

see osb #35
added sentry service for getting sentry DSN for applications
when an application is not yet know in Sentry it will be created
for convenience added cloud harness python library module function
utils.sentry.get_dsn
Usage examples:
  from cloudharness.sentry import get_dsn
  dsn = get_dsn('workspaces')

* small change

* removed bootstrap.sh.tar

* chore: refactoring utils/config.py

see #33
refactor so we have classmethods instead of instance methods
make CloudharnessConfig class public instead of private

* chore: added class comment to CloudharnessConfig class

see #33

* chore: add test if app has the harness.sentry setting set to True, if not or False then return '' indicating no Sentry DSN

see #35 osb

* fix: remove duplicate deployment element

* chore: refactor sentry to errormonitor, chservice to common, created cloudharness init, created sentry init to cloudharness init

see osb #35

* fix: fix reading sentry setting location, should be app.sentry instead of app.harness.sentry

see osb v2 #35

* fix: generate a comlete dsn instead of only the dsn public key

see osb v2 #35

* #37 Common service fixes

* add: new sentry service for capturing application events (logs, errors etc)

4 new pods:
sentry: main sentry web app
sentry-postgres-svr: postgres database for sentry
sentry-redis-svr: redis server for sentry
sentry-postfix-svr: mail MTA server for sentry (postfix)

for more info about sentry see https://sentry.io/

* Create Python api to retrieve infrastructure information

see #33
chore: added conf object to cloudharness utils, reworked env functions to use the conf object
  the new conf object is a simple wrapper around the allvalues.yaml
  it provides one function: get_application_by_filter
  returns app objects filtered by the filter
  example:
    conf.get_application_by_filter(harness__deployment__auto=True)
add: mount helm values.yaml to autodeploy applications
  add config map to transform the values into allvalues.yaml file
  mount the allvalues.yaml to the pod in /opt/cloudharness/resources/allvalues.yaml
add: added interface to access internal cloudharness allvalues dictionary

* chore: create new micro chservice api service for CH services

see osb #35
added sentry service for getting sentry DSN for applications
when an application is not yet know in Sentry it will be created
for convenience added cloud harness python library module function
utils.sentry.get_dsn
Usage examples:
  from cloudharness.sentry import get_dsn
  dsn = get_dsn('workspaces')

* small change

* removed bootstrap.sh.tar

* chore: refactoring utils/config.py

see #33
refactor so we have classmethods instead of instance methods
make CloudharnessConfig class public instead of private

* chore: added class comment to CloudharnessConfig class

see #33

* chore: add test if app has the harness.sentry setting set to True, if not or False then return '' indicating no Sentry DSN

see #35 osb

* fix: remove duplicate deployment element

* chore: refactor sentry to errormonitor, chservice to common, created cloudharness init, created sentry init to cloudharness init

see osb #35

* fix: fix reading sentry setting location, should be app.sentry instead of app.harness.sentry

see osb v2 #35

* fix: generate a comlete dsn instead of only the dsn public key

see osb v2 #35

* fix: add cors for common

see osb v2 #35

* Feature/37 Add sentry test (#42)

* #37 Update cli
* #37 added sentry test
* #37 added sentry api test

* #33 High level configuration api (#41)

* #33 High level application configuration api
* #33 Use new api on Sentry controller

* fix: changed some CH settings for sentry and sentry postgres

* osb # 46

* fix: add try catch around sentry init to overcome crashing of applications when sentry is not up/present

* see osb #46

* fix: changed postgres mount persistent volume to mount on /opt/data and initdb in /opt/data/pgdata because on GC the mounted volume isn't empty

* see osb #46

* Remove gatekeeper on samples

* Fix argo workflows python lib

* Improve command line tools and related documentation.

* #45 add persistent volume option to auto deploy apps (#44)

* chore: add persistent volume option to auto deploy apps

* chore: add support for persistent volume to auto deployment

* #46 increase proxy-body-size of Ingress server to 10M

* Generate client fix

* #50 Add parameter to specify included applications (#51)

* #50 Add parameter to specify included applications
* #50 Align include criterion with build

* #52 add namespace and domain to build args so it can be used during building of the images

* #52 fix typo

* #43 #57 JupyterHub custom authenticator and volume manager (#56)

* #43 connect jupyterhub to cloudharness keycloak

* #43 chore: some code cleanup and added namedServerLimitPerUser=10 config to values.yaml

* #57 chore: moved kubernetes pvc code from osb to ch

* #57 chore: added use of kubernetes config when not in a cluster

* #57 chore: added application hook

* #57 fix: when no user is provided logout user

* #57 chore: rm keycloakauthenticator

* #57 chore: change PVC accessmode to ReadWriteOnce

* #57 chore: removed LICENSE from chauthenticator

* #57 chore: removed ToDo flag and added comment on applicationHook usage

* #57 chore: moved volume manager from OSBv2 to CloudHarness

* chore: renamed some OSB references to CH

* #58 fix: changed keyUsage to nonRepudiation, digitalSignature, keyEncipherment (#59)

* #55 Application overriding introduced

* #54 Remove reference to other projects

* #55 Implement directory merge to build images

* #55 Use merge strategy

* #55 Instantiate docker client only on run

* #55 Fix merging

* #55 Remove / from registry in helm chart

* #55 Fix ingress for domain

* #55 Fix auto-gatekeeper

* #55 Fix dpaths typo in directorymerge operation

* #55 Fix dpaths typo in directorymerge operation

* #55 Debug CF pipeline

* #55 Test if CF is caching docker layers

* #54 Allow private keys as build-args

* #55 Polish PR

* #55 Polish PR

Co-authored-by: rodriguez-facundo <[email protected]>
Co-authored-by: Filippo Ledda <[email protected]>

* chore: added application specific secrets to deploy/values.yaml and cloudharness utilities

e.g. config
harness:
  ...
  secrets:
    - name: secret_1
      value: value_1
    - name: secret_2
      value: value_2
  ...

e.g. usage
from cloudharness.utils.secrets import get_secret

value_1 = get_secret("secret_1")
print(f"Secret 1 value: {value_1}"

* #62 Enable workflow tasks to mount an existing PVC  (merge PR #63)

* #63 chore: add zip extract to download task and use the default CH base image so all CH tools are present in the container

* #62 chore: enable mounting existing PVC in an argo workflow

* #62 fix: remove debug print() statement

* #61 chore: rework so application secrets need to be created on deployment and not on install

* #64 Bind gatekeeper to port 80, update docker image for gatekeeper

* Workflow fixes

* Default registry removed

* #66 fix: added argo-service to proxy to the argo ui and changed the argo port

* #66 chore: removed extra argo service entry and fixed argo values.yaml

* #67 fix: issue with events not working

* #68 fix: issue sentry handler not connecting to common rest api

* #69 chore: added EventClient.async_consume for async consuming events from a topic

* chore: changed loglevel to INFO

* #69 fix: fix issues, added more robust exception handling, added new image notify-queue for doing queue notification

* #70 SSL redirect option added

* #71 fix: replace double or more trialing slashes for registry to only one slash

* #71 fix: replace double or more trialing slashes for registry to only one slash

* Improved deploy robustness on missing applications

* #73 fix: add extra try except and sleep(10) to the thread when the kafka client is disconnected

* #78 Add option to disable tls (#79)

* #78 option to disable tls added
* #78 add tls option to accounts configuration

* #76 chore: added create kafka cluster on init

* Feature/78 (#81)

* #78 option to disable tls added
* #78 revert unrelated change
* #78 add tls option to accounts configuration
* #78 Fix no-http with jupyterhub
* #78 fix set auto gatekeeper secure cookie to false if tls is disabled

* Fix letsencrypt error

* fix letsencrypt

* fix letsencrypt

* #83 removed default reference to accounts application (#84)

* #82 chore: add uri role mapping for auto gk pods to (un)secure uri paths

* #86: added namespace to letsencrypt issuer name

* #86: removed doulbe - from ClusterIssuer name letsencrypt

* #88 add namespace to cluster default zookeeper hosts (#89)

* #90 make ingress listen to the naked domain

* #91 chore: added third parameter (folder) to the download extract task (#92)

* #94 chore: changed ClusterRole to Cluster and added namespace

* #100 fix: issue with filtering on getting applications by boolean filter True/False (#101)

* Feature/96 Support multiple paths in codefresh generate script (#98)

* #96 Support added to multiple directory in codefresh generator

* #102 chore: add get Keycloak user, group and users support to CH (#103)

* #102 chore: add get Keycloak user, group and users support to CH

* #102 fix: added missing HOST variable

* #102 chore: added get_current_user to get the current logged in user

* #97 New directory merge implementation (#104)

* #97 Directory merge do not change current deployment anymore
* #97 Fix build with merge
* #97 Fix build/codefresh with merge
* #97 Add create merge directory if does not exist
* #97 Merge copy preserving file attributes for better caching
* #97 Fix build include regression

* #105 chore: reworked keycloak client and added members to get_groups

* #105 fix: replaced ifn credentials/url by mnp credentials/url

* #105 chore: made get group and get admin client publicable

* #105 chore: added python-keycloak==0.22.0 to setup requirements

* #105 chore: reworked AuthClient to to use instance functions instead of static, added user has client role tests

* #105 fix: replaced wrong values for default realm, host and user in CH keycloak client

* #105 chore: some additions and bugfixes to CH keycloak library

* Feature/110 (#112)

* #110 Copy deployment-configuration with merge
* #111 run codefresh generation with harness-deployment

* Fix build parameter

* fix: replace .get[...] by .get(...)

* #113 chore: add keycloak create client and client role to ch common python lib

* Version update

* #115 chore: add/update/delete kc user attributes

* #115 fix: removed try except for add, update and delete user attributes

* #115 chore: add get users for client with a specific role

Co-authored-by: tarelli <[email protected]>
Co-authored-by: rodriguez-facundo <[email protected]>
Co-authored-by: Zoran Sinnema <[email protected]>
Co-authored-by: rodriguez-facundo <[email protected]>