Merge pull request #777 from MetaCell/release/2.4.0

Release/2.4.0

.dockerignore

@@ -19,4 +19,5 @@ __pycache__
 /application-templates
 /deployment-configuration
 /cloud-harness
-.openapi-generator
+.openapi-generator
+docker-compose.yaml

.github/workflows/lint-check.yml

@@ -0,0 +1,23 @@
+name: Lint Check
+
+on:
+  push
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.x'
+
+    - name: Install autopep8
+      run: pip install autopep8
+
+    - name: Run lint-check.sh
+      run: bash lint-check.sh

.gitignore

@@ -7,6 +7,8 @@ node_modules
 .coverage
 *.DS_Store
 deployment/helm
+deployment/compose
+docker-compose.yaml
 *.egg-info
 *.idea
 /build
@@ -17,4 +19,5 @@ skaffold.yaml
 /deployment.yaml
 .hypothesis
 __pycache__
-.env
+.env
+/.venv

README.md

@@ -2,12 +2,12 @@
 <img src="https://github.com/MetaCell/cloud-harness/blob/develop/cloudharness.png?raw=true" alt="drawing" width="200"/>
 </p>
 
-CloudHarness is a base infrastructure facilitator for microservice based applications deployed on Kubernetes.
+CloudHarness is a base infrastructure facilitator for microservice based applications deployed primarily on Kubernetes.
 Can scaffold and maintain your cloud solution on top of Cloudharness without writing
 Kubernetes templates, with in place common utilities and applications already configured for you.
 
 What building your cloud solution with CloudHarness gives to you:
-- Common framework and utilities to develop and deploy micro-service application 
+- Common framework and utilities to develop and deploy micro-service application
   - Helm chart automatic generation
     - deployments
     - services
@@ -17,6 +17,12 @@ What building your cloud solution with CloudHarness gives to you:
     - access gatekeepers configuration
     - secrets
     - templated config maps from files
+  - Docker compose configuration generation
+    - services
+    - traefik configuration
+    - databases (postgreql)
+    - access gatekeepers configuration
+    - secrets and configmaps
   * Automatic build and push of images
   * REST-API scaffolding building based on OpenApi
   * Continuous deployment script generation
@@ -46,17 +52,18 @@ In particular, these questions may rise:
  - How to manage databases without being locked to a specific vendor solution?
  - How to perform database backups?
  - How to manage secret data?
- - What about having a precounfigured account management application? 
- - Sooner rather than later I'll need an orchestration queue. Why not have that just ready to use? 
+ - What about having a precounfigured account management application?
+ - Sooner rather than later I'll need an orchestration queue. Why not have that just ready to use?
 
 # Command line tools
 
 CloudHarness provides the following command line tools to help application scaffolding and deployment.
 
-* `harness-deployment` - generate the helm chart to deploy on Kubernetes. 
+* `harness-deployment` - generate the helm chart to deploy on Kubernetes.
 * `harness-application` - create a new CloudHarness REST application.
 * `harness-generate` - generates server and client code for all CloudHarness REST applications.
 * `harness-test` - run end to end tests
+
 # Get started
 
 ## Prerequisites
@@ -67,15 +74,15 @@ Cloudharness can be used on all major operative systems.
 - Linux: supported and tested
 - MacOS: supported and tested
 - Windows/WSL2: supported and tested
-- Windows native: mostly working, unsupported  
+- Windows native: mostly working, unsupported
 
 ### Python
 Python 3.9 must be installed.
 
 It is recommended to setup a virtual environment.
-With conda: 
+With conda:
 ```bash
-conda create --name ch python=3.9
+conda create --name ch python=3.12
 conda activate ch
 ```
 
@@ -94,6 +101,10 @@ conda activate ch
 
 [Skaffold](https://skaffold.dev/docs/install/) is the way to go to build and debug your application in your local development environment.
 
+### Docker compose
+
+[Docker Compose](https://docs.docker.com/compose/) is required if the docker compose system is the target (instead of Kubernetes).
+
 ### Node environment
 
 A node environment with npm is required for developing web applications and to run end to end tests.
@@ -108,7 +119,6 @@ A JRE is needed to run the code generators based on openapi-generator.
 
 For more info, see [here](https://openapi-generator.tech/docs/installation).
 
-
 ## CloudHarness command line tools
 To use the cli tools, install requirements first:
 
@@ -127,24 +137,25 @@ To (re)generate the code for your applications, run `harness-generate` from the
 The script will look for all openapi applications, and regenerate the Flask server code and documentation.
 Note: the script will eventually override any manually modified file. To avoid that, define a file openapi-generator-ignore.
 
-# Extend CloudHarness to build your solution
-CloudHarness is born to be extended. In order to extend CloudHarness you just need to mirror the folder structure:
-* **applications**: place here your custom applications, or override default ones
-* **deployment-configuration**: override the helm chart default values and templates
-* **infrastructure**: define base images to use in your application
+# Extend CloudHarness to build your project
+
+CloudHarness is born to be extended. 
+
+The quickest way to start is to install Cloud Harness, copy the *blueprint* folder and build from that with the cli tools, such as
+`harness-application`, `harness-generate`, `harness-deployment`.
 
-or simply copy the *blueprint* folder.
+See the [developers documentation](docs/dev.md#start-your-project) for more information.
 
 # Build and deploy
 
 The script `harness-deployment` scans your applications and configurations to create the build and deploy artifacts.
 Created artifacts include:
- - Helm chart
+ - Helm chart (or docker compose configuration file)
  - Skaffold build and run configuration
  - Visual Studio Code debug and run configuration
  - Codefresh pipeline yaml specification (optional)
 
-With your solution folder structure looking like
+With your project folder structure looking like
 
 ```
 applications
@@ -153,7 +164,7 @@ infrastructure
 cloud-harness
 ```
 
-run 
+run
 
 ```
 harness-deployment cloud-harness . [PARAMS]

application-templates/base/test/api/test_st.py

@@ -3,7 +3,7 @@
 import schemathesis as st
 from schemathesis.checks import response_schema_conformance, not_a_server_error
 
-from cloudharness_test import apitest_init # include to perform default authorization
+from cloudharness_test import apitest_init  # include to perform default authorization
 
 app_url = os.environ.get("APP_URL", "http://samples.ch.local/api")
 
@@ -15,4 +15,3 @@ def test_ping(case):
     response = case.call()
     pprint(response.__dict__)
     assert response.status_code == 200, "this api errors on purpose"
-

application-templates/base/test/e2e/landing.spec.ts

@@ -29,7 +29,7 @@ describe("End to end test", () => {
       return el.textContent;
     });
 
-    expect(await page.title()).toEqual("Samples");
-    expect(title).toEqual("Sample React application is working!");
+    expect(await page.title()).not.toBeNull();
+    expect(title).not.toBeNull();
   });
 });

application-templates/django-app/api/templates/main.jinja2

@@ -14,13 +14,13 @@ from fastapi.staticfiles import StaticFiles
 
 {{imports | replace(".","openapi.")}}
 
-os.environ.setdefault("DJANGO_SETTINGS_MODULE", "__APP_NAME__.settings")
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "django_baseapp.settings")
 apps.populate(settings.INSTALLED_APPS)
 
 # migrate the Django models
 os.system("python manage.py migrate")
 
-from api.controllers import *
+from __APP_NAME__.controllers import *
 
 app = FastAPI(
     {% if info %}
@@ -49,34 +49,41 @@ app.add_middleware(
     allow_headers=["*"],
 )
 
-from cloudharness.middleware import set_authentication_token
+from cloudharness.middleware import set_authentication_token, get_authentication_token
 @app.middleware("http")
 async def add_process_time_header(request: Request, call_next):
     # retrieve the bearer token from the header
     # and save it for use in the AuthClient
-    authorization = request.headers.get('Authorization')
+    authorization = request.headers.get('Authorization') or request.cookies.get('kc-access')
+
     if authorization:
+        if 'Bearer ' in authorization:
+            authorization = authorization.split('Bearer ')[1]
+
         set_authentication_token(authorization)
 
     return await call_next(request)
 
 if os.environ.get('KUBERNETES_SERVICE_HOST', None):
     # init the auth service when running in/for k8s
-    from cloudharness_django.services import init_services, get_auth_service
-    init_services()
-    # start the kafka event listener when running in/for k8s
-    import cloudharness_django.services.events
+    from cloudharness_django.services import init_services_in_background, get_auth_service
+    init_services_in_background()
 
-# enable the Bearer Authentication
-security = HTTPBearer()
+    # start the kafka event listener when running in/for k8s
+    from cloudharness_django.services.events import init_listener_in_background
+    init_listener_in_background()
 
-async def has_access(credentials: HTTPBasicCredentials = Depends(security)):
+async def has_access():
     """
     Function that is used to validate the token in the case that it requires it
     """
     if not os.environ.get('KUBERNETES_SERVICE_HOST', None):
         return {}
-    token = credentials.credentials
+
+    token = get_authentication_token()
+
+    if not token:
+        raise HTTPException(status_code=401)
 
     try:
         payload = get_auth_service().get_auth_client().decode_token(token)

application-templates/django-app/api/test_st.py

@@ -3,7 +3,7 @@
 import schemathesis as st
 from schemathesis.checks import response_schema_conformance, not_a_server_error
 
-from cloudharness_test import apitest_init # include to perform default authorization
+from cloudharness_test import apitest_init  # include to perform default authorization
 
 app_url = os.environ.get("APP_URL", "http://samples.ch.local/api")
 
@@ -20,8 +20,9 @@ def test_ping(case):
     pprint(response.__dict__)
     assert response.status_code == 200, "this api errors on purpose"
 
+
 def test_state_machine():
     schema.as_state_machine().run()
 # APIWorkflow = schema.as_state_machine()
 # APIWorkflow.run()
-# TestAPI = APIWorkflow.TestCase
+# TestAPI = APIWorkflow.TestCase

application-templates/django-app/backend/api/apps.py → application-templates/django-app/backend/__APP_NAME__/apps.py

@@ -3,4 +3,4 @@
 
 class ApiConfig(AppConfig):
     default_auto_field = 'django.db.models.BigAutoField'
-    name = 'api'
+    name = '__APP_NAME__'

application-templates/django-app/backend/__APP_NAME__/controllers/__init__.py

@@ -0,0 +1 @@
+import __APP_NAME__.controllers.test as test_controller

application-templates/django-app/backend/__APP_NAME__/migrations/0001_initial.py

@@ -1,24 +1,10 @@
-import os
-
 from django.db import migrations
 
 
-def create_kc_client_and_roles(apps, schema_editor):
-    if os.environ.get("KUBERNETES_SERVICE_HOST", None):
-        # running in K8S so create the KC client and roles
-        from cloudharness_django.services import get_auth_service, get_user_service, init_services
-
-        init_services()
-        get_auth_service().create_client()
-        get_user_service().sync_kc_users_groups()
-
-
 class Migration(migrations.Migration):
 
     dependencies = [
-        ("cloudharness_django", "0001_initial"),
+        ("django_baseapp", "0001_initial"),
     ]
 
-    operations = [
-        migrations.RunPython(create_kc_client_and_roles),
-    ]
+    operations = []

application-templates/django-app/backend/__APP_NAME__/views.py

@@ -1,24 +1,3 @@
-import mimetypes
-from pathlib import Path
+from django.shortcuts import render
 
-from django.conf import settings
-from django.http import FileResponse, HttpResponseRedirect
-from django.urls import reverse
-from django.utils._os import safe_join
-
-
-def view_404(request, exception=None):
-    return HttpResponseRedirect(reverse("index"))
-
-
-def index(request, path=""):
-    if path == "":
-        path = "index.html"
-    fullpath = Path(safe_join(settings.STATIC_ROOT, "www", path))
-    content_type, encoding = mimetypes.guess_type(str(fullpath))
-    content_type = content_type or "application/octet-stream"
-    try:
-        fullpath.open("rb")
-    except FileNotFoundError:
-        return index(request, "")  # index.html
-    return FileResponse(fullpath.open("rb"), content_type=content_type)
+# Create your views here.

application-templates/django-app/backend/__APP_NAME__/asgi.py → application-templates/django-app/backend/django_baseapp/asgi.py

@@ -16,9 +16,11 @@
 application = get_asgi_application()
 
 # init the auth service
-from cloudharness_django.services import init_services
+from cloudharness_django.services import init_services  # noqa E402
 
 init_services()
 
 # start the kafka event listener
-import cloudharness_django.services.events
+from cloudharness_django.services.events import init_listner  # noqa E402
+
+init_listner()