Unable to use CA with a Helm deployment

[dr-lux]

Describe the bug

After a tweaked (more on that in How to reproduce) deployment of CA through Helm, I was able to log in. However in many main page content (like /analytics, /projects, ...) I got Internal Error 500.

From my logs I got this error django.db.utils.OperationalError: no such column: core_project.internal_reference

Here's the full logs of the ciso-assistant-backend in my Pod:

[2m2024-12-05T10:59:41.710137Z[0m [[32m[1minfo     [0m] [1mrequest_started               [0m [[0m[1m[34mdjango_structlog.middlewares.request[0m][0m [36mciso_assistant_url[0m=[35mhttps://<DOMAIN>[0m [36mip[0m=[35m127.0.0.1[0m [36mrequest[0m=[35mGET /api/iam/current-user/[0m [36mrequest_id[0m=[35mfeb2c693-c32b-410f-963d-c0d9e63a0bab[0m [36muser_agent[0m=[35mnode[0m [36muser_id[0m=[35mNone[0m
[2m2024-12-05T10:59:41.747670Z[0m [[32m[1minfo     [0m] [1mrequest_finished              [0m [[0m[1m[34mdjango_structlog.middlewares.request[0m][0m [36mciso_assistant_url[0m=[35mhttps://<DOMAIN>[0m [36mcode[0m=[35m200[0m [36mip[0m=[35m127.0.0.1[0m [36mrequest[0m=[35mGET /api/iam/current-user/[0m [36mrequest_id[0m=[35mfeb2c693-c32b-410f-963d-c0d9e63a0bab[0m [36muser_id[0m=[35m3e49b149-b4ce-45d1-bca9-eea958c2f33a[0m
[2m2024-12-05T10:59:41.751427Z[0m [[32m[1minfo     [0m] [1mrequest_started               [0m [[0m[1m[34mdjango_structlog.middlewares.request[0m][0m [36mciso_assistant_url[0m=[35mhttps://<DOMAIN>[0m [36mip[0m=[35m127.0.0.1[0m [36mrequest[0m=[35mGET /api/projects/[0m [36mrequest_id[0m=[35md8a8e55c-1851-4c7b-aa89-9a46c4447594[0m [36muser_agent[0m=[35mnode[0m [36muser_id[0m=[35mNone[0m
[2m2024-12-05T10:59:41.756890Z[0m [[31m[1merror    [0m] [1mrequest_failed                [0m [[0m[1m[34mdjango_structlog.middlewares.request[0m][0m [36mciso_assistant_url[0m=[35mhttps://<DOMAIN>[0m [36mcode[0m=[35m500[0m [36mip[0m=[35m127.0.0.1[0m [36mrequest[0m=[35mGET /api/projects/[0m [36mrequest_id[0m=[35md8a8e55c-1851-4c7b-aa89-9a46c4447594[0m [36muser_id[0m=[35m3e49b149-b4ce-45d1-bca9-eea958c2f33a[0m
Traceback (most recent call last):
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 354, in execute
    return super().execute(query, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such column: core_project.internal_reference

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/code/.venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/viewsets.py", line 124, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/mixins.py", line 38, in list
    queryset = self.filter_queryset(self.get_queryset())
                                    ^^^^^^^^^^^^^^^^^^^
  File "/code/core/views.py", line 103, in get_queryset
    object_ids_view = RoleAssignment.get_accessible_object_ids(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/iam/models.py", line 676, in get_accessible_object_ids
    folder_for_object = {x: Folder.get_folder(x) for x in all_objects}
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 400, in __iter__
    self._fetch_all()
  File "/code/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1574, in execute_sql
    cursor.execute(sql, params)
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/code/.venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 354, in execute
    return super().execute(query, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: no such column: core_project.internal_reference
[1mInternal Server Error: /api/projects/[0m
Traceback (most recent call last):
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 354, in execute
    return super().execute(query, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such column: core_project.internal_reference

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/code/.venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/viewsets.py", line 124, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/rest_framework/mixins.py", line 38, in list
    queryset = self.filter_queryset(self.get_queryset())
                                    ^^^^^^^^^^^^^^^^^^^
  File "/code/core/views.py", line 103, in get_queryset
    object_ids_view = RoleAssignment.get_accessible_object_ids(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/iam/models.py", line 676, in get_accessible_object_ids
    folder_for_object = {x: Folder.get_folder(x) for x in all_objects}
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 400, in __iter__
    self._fetch_all()
  File "/code/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1574, in execute_sql
    cursor.execute(sql, params)
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/code/.venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/.venv/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 354, in execute
    return super().execute(query, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: no such column: core_project.internal_reference

To Reproduce
Steps to reproduce the behavior:

1. As I use Traefik as our ingress, I added alongside your K8s object my own ingress and service to bypass the Nginx ingress and the Caddy container. In addition, I set the smtp values from my own HashiVault Secret. See the following templates output of my deployment

helm template -f charts/ciso-assistant/ciso-assistant/values-dev.yaml ciso-assistant charts/ciso-assistant

---
# Source: ciso-assistant/charts/cisoAssistant/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: smtp-out
  namespace: default
type: Opaque
data:
  EMAIL_HOST_PASSWORD: <HASHED_EMAIL_HOST_PASSWORD>
  EMAIL_HOST_PASSWORD_RESCUE: <HASHED_EMAIL_HOST_PASSWORD_RESCUE>
---
# Source: ciso-assistant/templates/vault-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: tls
  namespace: default
stringData:
  tls.crt: <HASHI_VAULT_SECRET_PATH_CRT>
  tls.key: <HASHI_VAULT_SECRET_PATH_KEY>
type: kubernetes.io/tls
---
# Source: ciso-assistant/templates/vault-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ciso-assistant-secret
  namespace: default
stringData:
  smtp-password: <HASHI_VAULT_SECRET_PATH_SMTP_PASSWORD>
  smtp-user: <HASHI_VAULT_SECRET_PATH_SMTP_USER>
---
# Source: ciso-assistant/charts/cisoAssistant/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ciso-assistant-config
  namespace: default
  labels:
    helm.sh/chart: cisoAssistant-0.1.0
    app.kubernetes.io/name: cisoAssistant
    app.kubernetes.io/instance: ciso-assistant
    app.kubernetes.io/version: "v1.9.7"
    app.kubernetes.io/managed-by: Helm
data:
  DEFAULT_FROM_EMAIL: "[email protected]"
  DJANGO_DEBUG: "false"
  EMAIL_HOST: "smtp.gmail.com"
  EMAIL_HOST_RESCUE: "smtp.secondary.mailer.cloud"
  EMAIL_HOST_USER: <HASHI_VAULT_SECRET_PATH_SMTP_USER>
  EMAIL_HOST_USER_RESCUE: "username"
  EMAIL_PORT: "587"
  EMAIL_PORT_RESCUE: "587"
  EMAIL_USE_TLS: "true"
  EMAIL_USE_TLS_RESCUE: "true"
  BODY_SIZE_LIMIT: "50000000"
---
# Source: ciso-assistant/charts/cisoAssistant/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ciso-assistant-ciso-assistant
  labels:
    client: ciso-assistant
    helm.sh/chart: cisoAssistant-0.1.0
    app.kubernetes.io/name: cisoAssistant
    app.kubernetes.io/instance: ciso-assistant
    app.kubernetes.io/version: "v1.9.7"
    app.kubernetes.io/managed-by: Helm
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: ciso-assistant
    client: ciso-assistant
  type: ClusterIP
---
# Source: ciso-assistant/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ciso-assistant-svc
  labels:
    client: 
    helm.sh/chart: ciso-assistant-0.1.0
    app.kubernetes.io/name: ciso-assistant
    app.kubernetes.io/instance: ciso-assistant
    app.kubernetes.io/version: ""
    app.kubernetes.io/managed-by: Helm
spec:
  ports:
  - name: frontend
    port: 3000
    protocol: TCP
    targetPort: 3000
  - name: backend
    port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    app: ciso-assistant
    client: 
  type: ClusterIP
---
# Source: ciso-assistant/charts/cisoAssistant/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ciso-assistant-ciso-assistant
  labels:
    app: ciso-assistant
    client: ciso-assistant
    version: v1.9.3
    helm.sh/chart: cisoAssistant-0.1.0
    app.kubernetes.io/name: cisoAssistant
    app.kubernetes.io/instance: ciso-assistant
    app.kubernetes.io/version: "v1.9.7"
    app.kubernetes.io/managed-by: Helm
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  selector:
    matchLabels:
      app: ciso-assistant
  serviceName: svc-ciso-assistant-ciso-assistant
  template:
    metadata:
      labels:
        app: ciso-assistant
        client: ciso-assistant
    spec:
      containers:
      - name: ciso-assistant-backend
        env:
        - name: CISO_ASSISTANT_SUPERUSER_EMAIL
          value: <CA_SUPERUSER_EMAIL>
        - name: CISO_ASSISTANT_URL
          value: https://<DOMAIN>
        - name: ALLOWED_HOSTS
          value: localhost,127.0.0.1,<DOMAIN>
        - name: EMAIL_HOST_PASSWORD
          valueFrom:
            secretKeyRef:
              key: EMAIL_HOST_PASSWORD
              name: smtp-out
        - name: EMAIL_HOST_PASSWORD_RESCUE
          valueFrom:
            secretKeyRef:
              key: EMAIL_HOST_PASSWORD_RESCUE
              name: smtp-out
        envFrom:
        - configMapRef:
            name: ciso-assistant-config
        image: "ghcr.io/intuitem/ciso-assistant-community/backend:v1.9.3"
        imagePullPolicy: Always
        ports:
        - containerPort: 8000
          protocol: TCP
        volumeMounts:
        - mountPath: /code/db
          name: db-data
      - name: ciso-assistant-frontend
        env:
        - name: ORIGIN
          value: "https://<DOMAIN>"
        - name: PUBLIC_BACKEND_API_EXPOSED_URL
          value: https://<DOMAIN>/api
        envFrom:
        - configMapRef:
            name: ciso-assistant-config
        image: "ghcr.io/intuitem/ciso-assistant-community/frontend:v1.9.3"
        imagePullPolicy: Always
        ports:
        - containerPort: 3000
          protocol: TCP
      - name: caddy
        command:
        - sh
        - '-c'
        - |
          echo <DOMAIN> "{" > Caddyfile
          echo "reverse_proxy /api/iam/sso/redirect/ localhost:8000" >> Caddyfile
          echo "reverse_proxy /api/accounts/saml/0/acs/ localhost:8000" >> Caddyfile
          echo "reverse_proxy /api/accounts/saml/0/acs/finish/ localhost:8000" >> Caddyfile
          echo "reverse_proxy /* localhost:3000" >> Caddyfile
          echo "}" >> Caddyfile
          exec caddy run
        env:
        - name: CISO_ASSISTANT_URL
          value: https://<DOMAIN>
        image: "caddy:2.7.6"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
          protocol: TCP
        - containerPort: 443
          protocol: TCP
        volumeMounts:
        - mountPath: /data
          name: db-data
          subPath: caddy
      enableServiceLinks: false
      imagePullSecrets:
      - name: registry-secret
      restartPolicy: Always
      volumes:
      - name: db-data
        persistentVolumeClaim:
          claimName: db-data
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      labels:
        app: ciso-assistant
        client: ciso-assistant
      name: db-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      volumeMode: Filesystem
---
# Source: ciso-assistant/charts/cisoAssistant/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: ciso-assistant-ciso-assistant
  labels:
    client: ciso-assistant
    helm.sh/chart: cisoAssistant-0.1.0
    app.kubernetes.io/name: cisoAssistant
    app.kubernetes.io/instance: ciso-assistant
    app.kubernetes.io/version: "v1.9.7"
    app.kubernetes.io/managed-by: Helm
spec:
  ingressClassName: nginx
  rules:
  - host: <DOMAIN>
    http:
      paths:
      - backend:
          service:
            name: ciso-assistant-ciso-assistant
            port:
              number: 443
        path: /
        pathType: Prefix
---
# Source: ciso-assistant/templates/ingress.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: <DOMAIN>
    external-dns.alpha.kubernetes.io/target: <TRAEFIK_DOMAIN>
    external-dns.alpha.kubernetes.io/ttl: "300"
    external-dns.alpha.kubernetes.io/zone: public
  name: cisoassistant-ing
spec:
  entryPoints:
    - websecure
  tls:
    secretName: tls
  routes:
    - kind: Rule
      match: Host(`<DOMAIN>`) && PathPrefix(`/`)
      services:
        - name: ciso-assistant-svc
          port: 3000
    - kind: Rule
      match: Host(`<DOMAIN>`) && PathPrefix(`/api/iam/sso/redirect`)
      services:
        - name: ciso-assistant-svc
          port: 8000
    - kind: Rule
      match: Host(`<DOMAIN>`) && PathPrefix(`/api/accounts/saml/0/acs`)
      services:
        - name: ciso-assistant-svc
          port: 8000
    - kind: Rule
      match: Host(`<DOMAIN>`) && PathPrefix(`/api/accounts/saml/o/acs/finish`)
      services:
        - name: ciso-assistant-svc
          port: 8000

2. Deploy this Chart
3. Log in
4. See in <DOMAIN>/analytics an error 500.

Expected behavior

Have Ciso Assistant works in K8s.

Screenshots

Environment

• Version v1.9.3 (default version from Ciso Assistant's Heml Chart)

Other

I think there is a lot of work to do with CA Helm Chart like:

• Proper container security
• Choice of Ingress (Prevent the Use of Let's Encrypt for Certificate Generation #1123)
• Split Pods by container for easier deployment
• Usage of proper storage object
• Usage of proper secret (the SMTP password is in plain text in values.yaml)

Thanks!

[dr-lux]

I facing the same issue with v1.3.16. I'm not using newer version (v1.3.17/18) because of Huey issue.