New lookup service code. (#527)

* Add initial files for workshop lookup service.

* Add handling of cluster configurations.

* Update HTTP error response codes for lookup service.

* Rename auth code file.

* Resource renaming and additional REST API endpoints.

* Drop unnecessary endpoint.

* Add means to register local cluster in lookup service.

* Add tracking of training portals.

* Capture portal auth details.

* Add tracking of workshops.

* Add lookup for portals hosting workshop.

* Renaming and start adding lookup of environments.

* Create derived view of portal and environment resources.

* Cross link actual cluster, portal and environment objects.

* Code restructuring.

* Fold environment database into portal.

* Fold portal database into cluster.

* Encapsulate REST API calls in portal.

* Start of trying to track sessions.

* Update to use user tracking for workshop sessions.

* Add additional details in workshop request response.

* Drop old code which is no longer required.

* Add scoring for candidate workshop environments.

* Add debugging for workshop environment selection.

* Update capacity details for workshop environment.

* Add logging for workshop requests.

* Code cleanup and add missing functionality.

* Split out routes registration from main program module.

* Add ability to query workshops by tenants endpoint.

* Add means to get portals from clusters.

* Align naming.

* Expand APIs for clusters.

* Simplify roles.

* Allow wildcards on tenant and cluster/portal selectors.

* Use environment variable for configuration namespace.

.gitignore

@@ -9,6 +9,7 @@ __pycache__
 /client-programs/bin
 /client-programs/pkg/renderer/files
 /developer-testing
+/lookup-service/venv
 /project-docs/venv
 /project-docs/_build
 /session-manager/venv

Makefile

@@ -24,14 +24,14 @@ build-all-images: build-session-manager build-training-portal \
   build-jdk17-environment build-jdk21-environment \
   build-conda-environment build-docker-registry \
   build-pause-container build-secrets-manager build-tunnel-manager \
-  build-image-cache build-assets-server
+  build-image-cache build-assets-server build-lookup-service
 
 push-all-images: push-session-manager push-training-portal \
   push-base-environment push-jdk8-environment push-jdk11-environment \
   push-jdk17-environment push-jdk21-environment \
   push-conda-environment push-docker-registry \
   push-pause-container push-secrets-manager push-tunnel-manager \
-  push-image-cache push-assets-server
+  push-image-cache push-assets-server push-lookup-service
 
 build-core-images: build-session-manager build-training-portal \
   build-base-environment build-docker-registry build-pause-container \
@@ -133,6 +133,12 @@ build-assets-server:
 push-assets-server: build-assets-server
 	docker push $(IMAGE_REPOSITORY)/educates-assets-server:$(PACKAGE_VERSION)
 
+build-lookup-service:
+	docker build --progress plain --platform $(DOCKER_PLATFORM) -t $(IMAGE_REPOSITORY)/educates-lookup-service:$(PACKAGE_VERSION) lookup-service
+
+push-lookup-service: build-lookup-service
+	docker push $(IMAGE_REPOSITORY)/educates-lookup-service:$(PACKAGE_VERSION)
+
 verify-installer-config:
 ifneq ("$(wildcard developer-testing/educates-installer-values.yaml)","")
 	@ytt --file carvel-packages/installer/bundle/config --data-values-file developer-testing/educates-installer-values.yaml

lookup-service/Dockerfile

@@ -0,0 +1,39 @@
+FROM fedora:39
+
+RUN INSTALL_PKGS=" \
+        findutils \
+        gcc \
+        glibc-langpack-en \
+        procps \
+        python3-devel \
+        python3-pip \
+        redhat-rpm-config \
+        which \
+    " && \
+    dnf install -y --setopt=tsflags=nodocs $INSTALL_PKGS && \
+    dnf clean -y --enablerepo='*' all && \
+    useradd -u 1001 -g 0 -M -d /opt/app-root/src default && \
+    mkdir -p /opt/app-root/src && \
+    chown -R 1001:0 /opt/app-root
+
+WORKDIR /opt/app-root/src
+
+ENV PYTHONUNBUFFERED=1 \
+    PYTHONIOENCODING=UTF-8 \
+    LC_ALL=en_US.UTF-8 \
+    LANG=en_US.UTF-8
+
+USER 1001
+
+COPY --chown=1001:0 requirements.txt /opt/app-root/requirements.txt
+
+ENV PATH=/opt/app-root/bin:/opt/app-root/venv/bin:$PATH
+
+RUN python3 -m venv /opt/app-root/venv && \
+    . /opt/app-root/venv/bin/activate && \
+    pip install --no-cache-dir -U pip setuptools wheel && \
+    pip install --no-cache-dir -r /opt/app-root/requirements.txt
+
+COPY --chown=1001:0 ./ /opt/app-root/src
+
+CMD [ "/opt/app-root/src/start-service.sh" ]

lookup-service/README.md

@@ -0,0 +1,6 @@
+Lookup Service
+==============
+
+This directory holds the source code for the Educates lookup service. It
+provides a high level REST API for accessing workshops, where workshops may
+be spread across one or more training portals, including across clusters.

lookup-service/requirements.txt

@@ -0,0 +1,7 @@
+kopf[full-auth]==1.37.2
+bcrypt==4.1.3
+aiohttp==3.9.5
+PyYAML==6.0.1
+pykube-ng==23.6.0
+wrapt==1.16.0
+PyJWT==2.8.0

lookup-service/service/caches/clients.py

@@ -0,0 +1,47 @@
+"""Configuration for clients of the service."""
+
+import fnmatch
+from dataclasses import dataclass
+from typing import List, Set
+
+
+@dataclass
+class ClientConfig:
+    """Configuration object for a client of the service."""
+
+    name: str
+    uid: str
+    password: str
+    tenants: List[str]
+    roles: List[str]
+
+    def check_password(self, password: str) -> bool:
+        """Checks the password provided against the client's password."""
+
+        return self.password == password
+
+    def validate_identity(self, uid: str) -> bool:
+        """Validate the identity provided against the client's identity."""
+
+        return self.uid == uid
+
+    def has_required_role(self, *roles: str) -> Set:
+        """Check if the client has any of the roles provided. We return back a
+        set containing the roles that matched."""
+
+        matched_roles = set()
+
+        for role in roles:
+            if role in self.roles:
+                matched_roles.add(role)
+
+        return matched_roles
+
+    def allowed_access_to_tenant(self, tenant: str) -> bool:
+        """Check if the client has access to the tenant."""
+
+        for pattern in self.tenants:
+            if fnmatch.fnmatch(tenant, pattern):
+                return True
+
+        return False

lookup-service/service/caches/clusters.py

@@ -0,0 +1,48 @@
+"""Configuration for target clusters."""
+
+from dataclasses import dataclass
+from typing import TYPE_CHECKING, Any, Dict, List
+
+if TYPE_CHECKING:
+    from .portals import TrainingPortal
+
+
+@dataclass
+class ClusterConfig:
+    """Configuration object for a target cluster. This includes a database of
+    the training portals hosted on the cluster."""
+
+    name: str
+    uid: str
+    labels: Dict[str, str]
+    kubeconfig: Dict[str, Any]
+    portals: Dict[str, "TrainingPortal"]
+
+    def __init__(
+        self, name: str, uid: str, labels: Dict[str, str], kubeconfig: Dict[str, Any]
+    ):
+        self.name = name
+        self.uid = uid
+        self.labels = labels
+        self.kubeconfig = kubeconfig
+        self.portals = {}
+
+    def add_portal(self, portal: "TrainingPortal") -> None:
+        """Add a portal to the cluster."""
+
+        self.portals[portal.name] = portal
+
+    def remove_portal(self, name: str) -> None:
+        """Remove a portal from the cluster."""
+
+        self.portals.pop(name, None)
+
+    def get_portals(self) -> List["TrainingPortal"]:
+        """Retrieve a list of portals from the cluster."""
+
+        return list(self.portals.values())
+
+    def get_portal(self, name: str) -> "TrainingPortal":
+        """Retrieve a portal from the cluster by name."""
+
+        return self.portals.get(name)

lookup-service/service/caches/databases.py

@@ -0,0 +1,126 @@
+"""Database classes for storing state of everything."""
+
+from dataclasses import dataclass
+from typing import TYPE_CHECKING, Dict, List
+
+if TYPE_CHECKING:
+    from .clients import ClientConfig
+    from .clusters import ClusterConfig
+    from .tenants import TenantConfig
+
+
+@dataclass
+class ClientDatabase:
+    """Database for storing client configurations. Clients are stored in a
+    dictionary with the client's name as the key and the client configuration
+    object as the value."""
+
+    clients: Dict[str, "ClientConfig"]
+
+    def __init__(self) -> None:
+        self.clients = {}
+
+    def update_client(self, client: "ClientConfig") -> None:
+        """Update the client in the database. If the client does not exist in
+        the database, it will be added."""
+
+        self.clients[client.name] = client
+
+    def remove_client(self, name: str) -> None:
+        """Remove a client from the database."""
+
+        self.clients.pop(name, None)
+
+    def get_clients(self) -> List["ClientConfig"]:
+        """Retrieve a list of clients from the database."""
+
+        return list(self.clients.values())
+
+    def get_client(self, name: str) -> "ClientConfig":
+        """Retrieve a client from the database by name."""
+
+        return self.clients.get(name)
+
+    def authenticate_client(self, name: str, password: str) -> str | None:
+        """Validate a client's credentials. Returning the uid of the client if
+        the credentials are valid."""
+
+        client = self.get_client(name)
+
+        if client is None:
+            return
+
+        if client.check_password(password):
+            return client.uid
+
+
+@dataclass
+class TenantDatabase:
+    """Database for storing tenant configurations. Tenants are stored in a
+    dictionary with the tenant's name as the key and the tenant configuration
+    object as the value."""
+
+    tenants: Dict[str, "TenantConfig"]
+
+    def __init__(self):
+        self.tenants = {}
+
+    def update_tenant(self, tenant: "TenantConfig") -> None:
+        """Update the tenant in the database. If the tenant does not exist in
+        the database, it will be added."""
+
+        self.tenants[tenant.name] = tenant
+
+    def remove_tenant(self, name: str) -> None:
+        """Remove a tenant from the database."""
+
+        self.tenants.pop(name, None)
+
+    def get_tenants(self) -> List["TenantConfig"]:
+        """Retrieve a list of tenants from the database."""
+
+        return list(self.tenants.values())
+
+    def get_tenant(self, name: str) -> "TenantConfig":
+        """Retrieve a tenant from the database by name."""
+
+        return self.tenants.get(name)
+
+
+@dataclass
+class ClusterDatabase:
+    """Database for storing cluster configurations. Clusters are stored in a
+    dictionary with the cluster's name as the key and the cluster configuration
+    object as the value."""
+
+    clusters: Dict[str, "ClusterConfig"]
+
+    def __init__(self) -> None:
+        self.clusters = {}
+
+    def add_cluster(self, cluster: "ClusterConfig") -> None:
+        """Add the cluster to the database."""
+
+        self.clusters[cluster.name] = cluster
+
+    def remove_cluster(self, name: str) -> None:
+        """Remove a cluster from the database."""
+
+        self.clusters.pop(name, None)
+
+    def get_clusters(self) -> List["ClusterConfig"]:
+        """Retrieve a list of clusters from the database."""
+
+        return list(self.clusters.values())
+
+    def get_cluster(self, name: str) -> "ClusterConfig":
+        """Retrieve a cluster from the database by name."""
+
+        return self.clusters.get(name)
+
+
+# Create the database instances.
+
+client_database = ClientDatabase()
+tenant_database = TenantDatabase()
+cluster_database = ClusterDatabase()